Vision-language models may not need to give words a place in the picture

When words inherit image coordinates

Vision-language models process words and image patches as tokens. Their attention mechanism lets tokens draw information from one another, while positional encoding supplies information about where those tokens belong.

Rotary Position Embedding, or RoPE, encodes position by rotating mathematical representations used to calculate attention. Multimodal RoPE extends this idea across temporal, height and width axes. Image patches receive coordinates reflecting their positions in an image grid.

Text tokens also receive coordinates in this shared system, despite having no inherent height or width in the picture. Consequently, a word’s attention to a patch can depend partly on their assigned spatial offset—not just their content. This does not mean the model always favors nearby patches, but it introduces a location-dependent influence.

Zhaoyi Liu and colleagues ask whether that influence helps connect language to visual regions, or adds an unnecessary complication.

Separating effects that normally come together

The researchers first establish a mathematical constraint. Under a shared coordinate assignment and the RoPE formulation they analyze, making positional rotations identical between a word and every patch in an image also makes the relative rotations between those patches trivial. Simply rearranging coordinates cannot completely remove one effect while retaining the other.

They also show why placing patches at equal geometric distances from words is insufficient: RoPE depends on rotations along individual axes, not just overall distance.

To separate the effects experimentally, the team uses different visual coordinates for different attention calculations. Patch-to-patch attention can retain the original image grid, while word–patch attention treats all patches within that image as having the same height and width coordinates.

This removes patch-dependent positional modulation from cross-modal attention. It does not force every patch to receive equal attention: their content representations can still differ, including information acquired through earlier processing.

The resulting experiment has four conditions: retain both spatial effects, remove either one independently, or remove both. Text coordinates and text-to-text attention remain unchanged.

Existing models barely needed the cross-modal effect

The team first changes attention calculations in released Qwen3-VL models with 2, 4 and 8 billion parameters, without retraining them.

Removing only the cross-modal spatial effect produces small changes in general visual understanding, spatial understanding and object localization. For the 4-billion-parameter model, localization accuracy at an overlap threshold of 0.5 changes from 87.1% to 87.0%.

Removing spatial encoding between image patches has a dramatically different effect. At that same threshold, accuracy falls from 87.1% to 16.3%. The other model sizes also suffer large localization losses.

Here, overlap means intersection-over-union: the area shared by a predicted bounding box and the reference box, divided by their combined area. Higher thresholds demand tighter localization.

These inference-time interventions suggest that the tested checkpoints rely heavily on within-image spatial encoding, but little on the additional cross-modal positional differences.

Training without the effect sometimes helps

An inference-time change cannot reveal whether a model spent training learning to accommodate an unhelpful influence. The researchers therefore run controlled training experiments at the 2-billion- and 4-billion-parameter scales.

They reuse pretrained vision encoders and language backbones, but randomly initialize the components connecting them. Within each scale, all four conditions start with identical parameters and follow the same two-stage training procedure: first train the connecting components, then fine-tune the whole model with visual instruction and grounding data.

For the main MRoPE-I experiments, results are reported across three random seeds. Removing cross-modal spatial effects raises average spatial-understanding scores from 34.9 to 36.1 for the smaller model and from 38.0 to 39.3 for the larger one. General visual-understanding averages remain similar.

Precise grounding also improves. At the demanding 0.9 overlap threshold, accuracy rises from 19.3% to 20.5% and from 28.0% to 29.2%—a gain of 1.2 percentage points at each scale.

Experiments with VideoRoPE, HoPE and IL-RoPE report similar benefits for precise grounding, although those extensions use a single seed rather than the main experiment’s three.

A design clue, not a universal replacement

The distinction matters: removing within-image spatial encoding during controlled training does not reproduce the catastrophic losses seen when changing already-trained checkpoints. Models trained under altered conditions can adapt; the two experiments answer different questions.

The evidence also remains concentrated in the Qwen model family, image-based benchmarks and comparatively limited training regimes. It does not establish equivalent results for video, other architectures or substantially larger training runs. Nor does the study establish a deployment-speed advantage; its implementation uses eager attention.

The November 9 arXiv preprint supports a narrower conclusion: preserving an image’s internal spatial relationships need not require imposing patch-dependent positional effects on its interactions with language. That is a useful distinction for future model design, not evidence that spatial information itself is dispensable.