VagueBench tests why embodied AI agents struggle with shorter instructions

An embodied AI agent must connect language to observations and actions: identify an object, decide what to do, and carry out a sequence that reaches the requested outcome. But how much does its success depend on instructions spelling out details that are not essential to the task?

In the preprint VagueBench: Benchmarking the Robustness of Embodied Agents on Ambient Context Illusions, Yang Qu and colleagues investigate that question by comparing detailed instructions with context-reduced counterparts. They call the resulting failure pattern “Ambient Context Illusion”: an agent behaves incorrectly after auxiliary details are removed, even though the task remains well-defined in its environment.

The distinction matters. The intended test is not whether agents can resolve genuinely ambiguous requests. It is whether they can execute instructions judged sufficient without extra linguistic guidance.

Constructing the comparison

VagueBench contains 1,303 instruction pairs for indoor tasks in the AI2-THOR simulator. The instructions originate from ALFRED, a dataset of language-guided embodied tasks. Scenes include kitchens, living rooms, bedrooms and bathrooms; kitchens account for 49% of samples.

The construction process combines automated proposals with human review. An agent estimates which contextual information could be removed, human annotators check those candidates, and transformations produce shorter counterparts. Three expert annotators then independently assess each pair. Only pairs unanimously judged to preserve the core task intent and remain unambiguously executable in the environment are retained.

The transformations cover four categories and nine subtypes. They include removing object attributes, compressing procedural steps, relaxing spatial descriptions, and omitting cues associated with prior knowledge or observations. One example replaces a detailed sequence involving retrieving an apple from a microwave and rinsing it in a sink with “Get the apple, rinse it, and microwave it.” Individual pairs can involve multiple transformation types.

Semantic equivalence is therefore a scene-specific judgment in this benchmark, not a claim that the shortened phrases would identify the same task in every setting. The paper also describes constraint relaxation as enlarging the feasible solution space, making its human verification of preserved intent especially important.

Performance falls with less context

The main evaluation covers four vision-language models and two agent workflows, ProgPrompt and ReAct. The researchers measure complete task success, partial subgoal completion, planner errors, invalid actions and empty plans.

All six evaluated configurations show lower task success and subgoal completion with context-reduced instructions. For Qwen2.5-VL-72B-Instruct, task success falls from 43.51% to 28.40%: a decrease of 15.11 percentage points. GPT-4o falls from 49.12% to 39.52%, a decrease of 9.60 percentage points.

Execution-related results are less uniform. Empty plans become more frequent in five configurations, while ReAct's rate remains zero. ProgPrompt records fewer planner errors and a lower invalid-action ratio despite completing fewer tasks. These exceptions matter: producing fewer obvious execution errors does not necessarily mean an agent is accomplishing the requested goal more reliably.

The authors interpret the overall pattern as evidence of reliance on auxiliary linguistic cues. Their more specific explanations—for example, that entity descriptions anchor object identification while procedural details support task decomposition—are hypotheses about the observed failures, rather than independently established mechanisms.

Physical tests and a mitigation trade-off

The researchers also test a myCobot 280-Pi robotic arm controlled by Qwen2.5-VL-72B-Instruct. In an earphone-moving case study evaluated over ten independent trials, reported success falls from 50% with the detailed instruction to 10% with its shortened counterpart.

An expanded physical evaluation uses three instruction pairs per transformation subtype, totaling 54 instructions and 270 executions. Reported success falls from 77.78% to 45.93%. These controlled experiments support the presence of the problem outside simulation, but do not establish its prevalence across deployed robots or demonstrate actual safety incidents. The authors note that the physical tasks are simpler than those in simulation.

Their proposed mitigation, Context Suppression Constraint, adds inference-time guidance rather than changing model parameters. It prioritizes the target object and desired final state, discourages unsupported assumptions, and suppresses unnecessary contextual information.

On Qwen2.5-VL-72B-Instruct, the relative success-rate drop between detailed and reduced instructions shrinks from 34.73% to 11.29%. However, success with detailed instructions also declines, from 43.51% to 39.98%. The improvement in consistency comes with a measurable trade-off.

VagueBench offers evidence that concise, human-validated instructions deserve explicit robustness testing. The preprint leaves broader environments, more complex robotic systems and learning-based alternatives to prompt constraints as future work.