BEAKER benchmark reveals a gap between understanding lab scenes and locating where to act

A model may recognize laboratory equipment or follow an experimental video without identifying exactly where a machine should interact with the apparatus. That distinction motivates BEAKER, a benchmark introduced by Fei Lin, Tengchao Zhang, Ziyang Gong and colleagues in an arXiv preprint submitted on November 4, 2026. The researchers evaluate multimodal large language models as potential high-level perception and reasoning systems for self-driving chemical laboratories—not as demonstrated autonomous laboratory operators.

Across 28 models, the reported pattern is consistent: coarse procedural understanding is stronger than precise spatial grounding. The benchmark measures these capabilities through questions about recorded images and videos, rather than through physical experiments performed by robots.

Questions designed around laboratory execution

BEAKER organizes 11 task types into four dimensions: Actionability, Knowledge, Experimentation and Reasoning. Questions cover locating operable regions, proposing trajectories, identifying equipment and reagent states, recognizing safety norms, understanding apparatus setups and tracking experimental steps.

The final dataset contains 600 image-based and 400 video-based questions. Answers include image coordinates, sequences of trajectory waypoints, multiple-choice selections and domain-specific phrases completing blanks. Spatial questions target functional regions such as valve handles and pipette nozzles, rather than merely asking models to recognize whole objects.

The construction process involved 10 chemistry experts and 12 embodied-AI experts. They produced 1,500 candidate questions, with cross-group blind review and consensus adjudication removing 500. The retained questions draw on 410 images and 400 video clips.

The authors emphasize that experts selected samples and wrote and reviewed questions and answers without AI-generated annotations. There is a narrower technical qualification: Whisper supplied speech timestamps to help propose video-segment boundaries. It did not select the retained samples or author their questions and answers.

Where the models struggled

All models received standardized, task-specific prompts in an offline, single-turn evaluation. Videos were presented as sampled frames: eight for short clips, 32 for mid-length clips and 128, arranged into four-frame grids, for long recordings. Three chemistry experts uninvolved in benchmark construction independently answered all questions to establish a human baseline.

Claude Opus 4.7 achieved the highest reported overall model score, 59.2 out of 100, compared with 93.3 for the human baseline—a difference of 34.1 points. These are composite benchmark scores, not percentages of successful laboratory operations. They combine multiple-choice accuracy, spatial-overlap and trajectory-distance measures, and expert assessments of fill-in-the-blank answers using fixed, expert-defined weights.

Across all 28 models, Actionability averaged 22.2, compared with 41.6 for Knowledge, 58.0 for Experimentation and 53.8 for Reasoning. On affordance grounding—locating the precise region relevant to an interaction—the mean score was just 9.7. This metric measures overlap between reference boxes and equally sized boxes centered on model-predicted points; it does not measure grasp success.

Models performed better on short-video operation reasoning, averaging 65.7. However, ordering steps in mid-length videos remained difficult: the corresponding temporal-sequence score averaged 37.1, versus 68.8 for short videos and 57.0 for long videos. The authors interpret this as difficulty extracting fine-grained operations and step boundaries, rather than a simple decline with increasing video length.

What additional context revealed

Providing expert-written captions substantially improved video-task performance. For mid-length temporal ordering, caption-only scores rose to 92–96 across three tested models, compared with 44–48 using video frames alone. This supports the authors’ interpretation that extracting procedural information from visual inputs is an important bottleneck. It does not establish that captions resolve autonomous execution.

Increasing frame counts also failed to improve every task consistently in an experiment with one model. More visual input was therefore not a universal remedy under the tested settings.

Chemistry-specialized and science-oriented models did not consistently outperform general-purpose models. The paper cautions against attributing this to specialization alone: different underlying models and training recipes prevent a controlled causal comparison.

A diagnostic, not deployment certification

BEAKER does not test closed-loop control, multi-turn interaction or recovery from errors. Its coordinates and trajectories approximate action-related judgments offline, and the study reports no correlation between benchmark scores and real-robot execution success.

Coverage is also limited, and public-source images and videos may have appeared in model training data, although the benchmark questions and annotations were newly authored. Its composite score combines unlike metrics, making individual task profiles important for interpretation.

The researchers propose future simulation-based evaluation in which actions are executed and models respond to feedback. For now, BEAKER identifies gaps in laboratory perception and reasoning; it does not certify models for autonomous chemical work.