MESA tests where robot skills stop generalizing

Robotic manipulation benchmarks often compress different capabilities into one success score. MESA instead asks more specific questions: can a robot recognize an unfamiliar object, adapt to a new location, or combine familiar actions into an unfamiliar task? Albert Wilcox, Frank Chang and colleagues at Georgia Tech introduce the framework in a paper posted to arXiv on November 3, 2026. The manuscript and landing page identify it with the Conference on Robot Learning, CoRL 2026.

The work combines MESA-Gen, a task and demonstration generation pipeline, with MESA-Bench, a fixed simulation benchmark. It includes a single-arm Franka Panda setup and a two-arm YAM setup called BiMESA. Its purpose is controlled diagnosis rather than a comprehensive test of everything a useful robot should do.

Five suites separate different challenges

Each embodiment has one training dataset and five evaluation suites. The in-distribution suite tests familiar tasks with newly sampled scenes and distractors. Four additional suites probe specific changes:

• Spatial: familiar tasks with held-out object locations or arm configurations.
• Instance: familiar object categories represented by unseen object assets.
• Category: objects from categories absent from benchmark training.
• Composite: unfamiliar combinations of familiar subtasks, including some longer action sequences.

The bimanual spatial tests go beyond moving an object across a table. They include held-out combinations of picking and destination sides, reversed handoff directions, and articulated fixtures placed on a different side. These changes test whether a policy can reuse skills across arms rather than reproduce a learned arm-specific motion.

Distractors are central to the design. Scenes contain multiple plausible object-and-destination combinations, so looking at the scene alone should not reveal the intended task. In an ablation, randomizing instructions sharply reduced the strongest single-arm baseline’s performance, supporting the authors’ claim that the benchmark requires language conditioning.

Generating demonstrations from reusable pieces

The main single-arm training set contains 7,000 demonstrations across 70 tasks. BiMESA supplies 6,833 demonstrations across 57 tasks, mixing generated trajectories with human teleoperation. Coordinated cross-arm motions and handoffs require teleoperated data in this implementation.

MESA-Gen extends MimicGen by stitching together object-centered subtasks from different source demonstrations. A grasp taken from one task can be combined with a placement taken from another, allowing demonstration generation without a human example of every complete task. Trajectory adjustments also aim to reduce unnecessary sweeping motions and self-collisions when object positions differ substantially from the source scene.

These improvements are not a general solution to robot data generation. The authors acknowledge that the pipeline struggles with dense clutter and can still produce suboptimal trajectories.

Strong models still leave substantial gaps

The study evaluates seven baseline methods spanning a multitask diffusion policy, vision-language models adapted for control, and pretrained vision-language-action models. One baseline, PG-Bin, is reported only for BiMESA because an inference bug affected its single-arm evaluation.

The strongest overall baseline, π0.5, averaged 52.0% success across the five single-arm suites and 48.7% across the five bimanual suites. These are suite averages, not scores from physical robots. For π0.5, success on unfamiliar object categories was 26.4% in the single-arm setting and 18.6% in BiMESA. GR00T-N1.6 performed better on BiMESA’s category suite, reaching 26.1%, despite its lower overall average.

A separate zero-shot GPT-6.1-Sol agent reached 52.4% on BiMESA’s category suite. That comparison needs qualification: the agent used an end-effector tool-calling harness and received 10 rollouts per task rather than the baselines’ 50. Its median latency was 13 times simulated robot time, so it did not operate in real time.

Measuring instruction errors without explaining every failure

MESA’s language-following rate divides intended-task successes by the combined total of intended-task successes and failed rollouts that complete a distractor task. This distinguishes completing the wrong goal from completing the requested one, conditional on completing one of those goals.

It does not explain episodes in which neither goal is completed. Those may reflect misunderstanding, execution failure, or both. The metric also does not capture partial completion.

Broader limits remain. Tasks mainly involve pick-and-place and simple articulated manipulation, not dexterous, mobile or broadly long-horizon behavior. Statistical intervals describe rollout variation for one trained checkpoint per method, not variation across training seeds or task selection.

Rankings were broadly consistent with RoboArena’s real-world evaluations, but those comparisons involved differently trained policies. Real-trained checkpoints tested without MESA finetuning produced too little success for reliable rankings. The authors therefore present real-world relevance as preliminary evidence, not established transfer.