Virtual shopping benchmark tests whether AI can recommend, listen and guide

Shopping as an interactive task

Jingtong Yang, Dongding Lin, Jian Wang and Wenjie Li developed SituRecBench to investigate whether AI assistants can combine conversation, recommendations and spatial guidance. Their paper is available as an arXiv preprint; its landing page reports acceptance to the NeurIPS 2026 Evaluations and Datasets Track.

The benchmark evaluates vision-language models: AI systems that process images and text. Here, an assistant shares the shopper’s egocentric view, meaning the scene seen from the shopper’s first-person perspective. It must connect that local view with broader information about the store while learning preferences through conversation.

Unlike a conventional product-ranking test, each exchange can change what happens next. A shopper might follow directions, inspect an item, reject a suggestion or reveal another requirement. The assistant must respond to those developments rather than simply produce a list.

How the virtual stores work

The researchers built a supermarket and a home-furnishings store in Unreal Engine 5. Together, they contain 455 distinct catalog products. The benchmark includes 50 shopper profiles and 200 shopping tasks, divided equally between the environments.

In the main experiments, a model plays the shopper using a private profile, shopping goal, preferences and budget. The assistant does not receive those private requirements directly. Instead, it receives a product catalog, location information, a map showing the shopper’s position and orientation, and annotated first-person observations.

Visual markers identify products and navigation targets. The shopper and assistant receive different marker labels, preventing them from coordinating simply by repeating the same identifiers.

The assistant communicates guidance; the simulated shopper independently selects a physical action, such as turning, moving or inspecting a product. The environment then updates. Episodes end when an item enters the cart or another stopping condition occurs, including a 20-turn limit or three recommendation attempts without a purchase.

Seven assistant models each completed all 200 tasks, producing 1,400 interaction records. GPT-5.6 Sol served as both the simulated shopper and the dialogue judge in these main experiments.

Different strengths, different outcomes

The researchers measured purchase completion, dialogue quality, recommendation rankings and how closely shoppers followed guidance. These measures capture different things: following an instruction does not necessarily mean finding a suitable product.

Gemini 3.1 Pro achieved the highest cart-addition rate in both environments: 80% in the supermarket and 66% in the furnishings store. GPT-5.6 Sol reached 69% and 65%, respectively, while recording the highest action-following rates. Gemini also led the main recommendation-ranking measures, which compare ranked suggestions with products eventually purchased.

Those ranking measures are not independent assessments of suitability: they use the final cart as their reference, and unsuccessful episodes score zero. A separate check compared purchases with products meeting task requirements and budgets. For Gemini, such purchases occurred in 74% of supermarket tasks and 54% of furnishings tasks—below its cart-addition rates.

Individual examples illustrate the problems behind the totals. In one unsuccessful supermarket interaction, GPT-4o recommended fish costing $10.99 after the shopper explicitly stated a $10 limit. Furniture examples showed the same chair accepted for reading but rejected for its appearance in a warmer-looking interior. Matching a product category alone was insufficient.

What the visual-input test revealed

An ablation study—an experiment removing parts of the input—tested GPT-5.6 Sol on 100 supermarket tasks per condition. Removing first-person images while retaining textual descriptions of visible targets left purchase outcomes broadly similar, although navigation alignment declined.

Removing those descriptions too reduced purchase and recommendation scores and all dialogue ratings. This suggests the textual annotations helped connect the global catalog and map with nearby objects. It does not establish that images are generally unnecessary: the experiment retained substantial structured information, and the joint-removal condition also changed some action constraints.

Limits of the evidence

Human validation involved five participants and 50 supermarket interactions across five assistants. Human and simulated evaluations showed similar relative performance on some measures, including overall satisfaction and local navigation alignment, but agreement was not uniform. The text-only dialogue judge could not directly verify visual details, which the authors suggest may explain weaker agreement on groundedness.

The environments also provide still first-person views at each turn, rather than continuous video during movement. These are controlled virtual shopping tests, not evidence of reliable assistance in real stores. The results identify a continuing challenge: coordinating what an assistant knows about the whole environment with what the shopper currently sees, wants and actually does.