PhysMoE gives robot vision-language models a more direct way to point

For a robot, understanding an instruction is only part of the job. It also has to connect words such as “grasp the handle” to a location in an image. That process, called spatial grounding, can be awkward for vision-language models (VLMs): coordinates written as text may be split into several tokens, while predicting one point can be a poor fit when many points on a region would work.

A preprint by Ruimeng Liu and colleagues introduces PhysMoE, a model design intended to address both problems. The authors evaluate it on tasks including pointing to affordances—parts of objects suited to an action—finding locations, and predicting trajectories. They also test whether changing the model for coordinate tasks harms its broader text-based embodied reasoning.

A coordinate as one prediction

Instead of producing a coordinate as a string of digits and punctuation, PhysMoE represents a two-dimensional coordinate as a single unit and predicts its continuous image position directly. To represent that position inside the model, it reuses learned positional information from the visual encoder: in simple terms, a map the vision system already uses to represent where image patches are.

The model also divides some of its processing between language and spatial experts. Earlier layers remain shared, so text and image information can be combined. In the upper layers, specialized components handle language or coordinate outputs. The design aims to keep spatial training from disrupting language processing while retaining shared context.

For regions where several contact points are valid, the authors use a signed distance field (SDF) loss. This represents how far a predicted point is from the boundary of an annotated region, and whether it lies inside or outside. Unlike training toward a single chosen point, this approach can reward predictions anywhere within a usable region. The training masks were created from existing annotations with an automated segmentation model, then converted into distance maps.

What the authors report

The paper evaluates two PhysMoE model sizes on 20 benchmarks: nine involving coordinate-based grounding and 11 involving text-based embodied understanding or reasoning. On the coordinate tasks, the reported average score rises from 56.3 to 67.8 for the 4-billion-parameter model compared with its starting Qwen3.5 model, and from 64.1 to 74.2 for the 9-billion-parameter model compared with its starting RynnBrain model. These are comparisons with each model’s own base, not a single uniform comparison across all systems.

For text-based tasks, the reported average is 62.7 for PhysMoE-4B versus 61.3 for Qwen3.5-4B, and 70.9 for PhysMoE-9B versus 71.4 for RynnBrain 1.1. In other words, the scores are close, but they do not show an improvement at both sizes.

The authors also measured coordinate-output throughput on an NVIDIA RTX 5090 using 1,000 sampled question-answer instances. PhysMoE-9B produced 23.34 samples per second, compared with 2.81 for the listed Embodied-R1.5-8B baseline; PhysMoE-4B produced 31.88, compared with 5.28 for RynnBrain1.0-4B. These figures concern the paper’s defined coordinate-output task and hardware setup. They are not a general measure of robot speed or end-to-end task success.

What remains uncertain

The evidence is from benchmark evaluations, not a demonstration that PhysMoE makes robots safer or more reliable in everyday environments. Image-coordinate accuracy does not by itself establish that a robot can execute a grasp, cope with changing scenes, or recover from errors. The paper describes a large training mixture and use of automatically generated masks, so results may also depend on the data and annotation pipeline as well as the architecture.

The authors report that their method preserves text-based performance, but benchmark averages cannot establish that all kinds of reasoning are unaffected. Comparisons also use different starting models at the two sizes, and some competing-model results are adopted from another paper rather than evaluated locally under the same setup, according to the methods section.

This article is based on the supplied arXiv preprint text, not an independent replication or a peer-reviewed assessment. The paper’s claims should therefore be read as reported results that need further scrutiny. PhysMoE offers a specific engineering proposal—make coordinates more direct to represent and allow valid regions rather than only single points—but whether that translates into more capable physical robots remains an open question.