Why image-search models miss left and right—and a targeted way to recover them

A search system can recognize both objects in a photograph yet fail to distinguish their arrangement. In a study of ten vision–language encoders, Mahdi Mostajabdaveh and Zirui Zhou of Huawei Technologies Canada argue that a near-chance left/right score does not reveal where the failure happened.

Their paper, Discarded or Unreachable? Diagnosing and Repairing Spatial Relation Failure in Vision–Language Similarity, was submitted to arXiv on November 3, 2026. Its landing page lists WACV 2027 as the journal reference. The authors distinguish two failures and propose a training-free readout that addresses both in their experiments.

Information present, but not accessed

The models studied encode an image and a caption separately, then compare their vectors using an inner product. On the image side, a grid of local features is pooled into the single vector typically stored for retrieval.

The researchers first examined clean, two-object photographs from the What’s Up benchmark. They compared each image’s pooled vector with the vector of its horizontal mirror, isolating the component that changes sign under reflection. Supervised linear probes recovered left/right polarity from this component with accuracies of 77.1%–99.0% across the ten encoders.

Standard caption-based scoring, however, remained near chance under ranking protocols designed to cancel fixed caption preferences and control for preferences for unflipped photographs. The authors call this an access failure: a trained reader can recover information that ordinary image–text similarity does not use successfully.

That conclusion has an important boundary. In these photographs, a smaller object moves beside a nearly stationary central reference object. Its position alone usually determines the relation label. The probe results therefore do not establish that the pooled vector binds two objects to their respective locations.

Nor does the caption search prove that language can never access the information. Searching a million captions left nine encoders near chance, but found a successful caption pair for ConvNeXt-CLIP, with median held-out accuracy of 67.8% across 20 splits.

Information lost during pooling

A second experiment used 300 controlled four-image sets. Two segmented objects occupied positions around a fixed centre; their displacement rotated in 90-degree steps while the object pixels stayed upright. This separated the arrangement’s axis—horizontal or vertical—from its polarity—which object was on which side.

The pooled vectors supported axis classification at 99.8%–100.0%, but polarity was poorly recovered. Nonlinear probes generally did not resolve the deficit, although RN50-CLIP showed partial exceptions. Meanwhile, a readout from the patch grid identified the full four-way relation at 81.1%–98.6%.

The authors interpret this as pooling failure. Their evidence concerns the readers, samples and stimuli tested; it is not a proof that every possible decoder would fail.

Recovering geometry without retraining

The proposed repair returns to the local features before pooling. Using the encoder’s own pooling head, it derives patch features in the shared image–text space, then computes coordinate-weighted averages along the horizontal and vertical axes.

These two “spatial moments” are stored beside the native image vector. A parsed query such as “a dog left of a table” supplies the object names, axis and sign. Retrieval remains an inner-product calculation over augmented vectors, without requiring the patch grid at query time.

The method adds no learned parameters. Its spatial weight is estimated from unlabelled score statistics. When the spatial query channels are zero, the native score is preserved mathematically; the paper also checks floating-point agreement and tie-aware ordering.

Strong conditional results, narrower deployment scope

Using an index of 118,287 COCO images, the researchers tested queries against distractors containing the same two object categories but reversed geometry. Left/right pairwise accuracy rose from 51.0%–52.7% to 80.8%–96.2%. Among 21 category-matched candidates, top-rank accuracy rose from 4.5%–5.9% to 36.5%–75.3%.

Those are conditional results, not success rates for unrestricted search across the entire index. The reported sub-millisecond batched timings cover scoring against cached image vectors and exclude query-text encoding. Uncompressed storage triples; a PCA sketch reduces that overhead.

Query understanding remains a bottleneck. On generated free-form horizontal queries, the simple parser supplied a triple for about 37%; fallback to native scoring helped reduce mean end-to-end pairwise accuracy to 62.8%, versus 87.4% with triples supplied.

More decisively, placing the parser before standard COCO and Flickr30k caption retrieval lowered overall top-rank recall by roughly 0.5–1.5 percentage points. The spatial term helped distinguish arrangements when object categories were held fixed, but disrupted broader ranking competition.

The method also needs internal features, cannot disambiguate multiple instances of one category through a single location estimate, and does not establish results for front/behind relations. The contribution is a targeted diagnosis and repair—not a general solution to spatial understanding.