AtariBench exposes AI models’ difficulty turning moving images into reliable actions

From recognizing scenes to controlling them

Vision-language models can interpret pictures and respond to written instructions, but interactive environments demand more: tracking what changes, anticipating movement and choosing actions that fit the latest observation. AtariBench puts those abilities under controlled scrutiny across 20 Atari 2600 games.

Mingjia Huo and colleagues at the University of California San Diego, Google DeepMind, Peking University and the University of Illinois Urbana-Champaign describe the benchmark in an arXiv manuscript submitted on November 5, 2026. Its listing states that the paper has been accepted to the NeurIPS 2026 Evaluations and Datasets Track.

The researchers report a persistent gap between understanding visual scenes and acting reliably within them. Their findings also challenge two seemingly straightforward remedies: giving models more interaction history or asking them to reason more extensively.

A controlled test, not a reaction-speed contest

AtariBench runs on the Arcade Learning Environment. Its games cover shooting, sports, action and maze navigation, including Breakout, Freeway and Pac-Man. The team selected games requiring motion perception, with controls and scoring that should be understandable within a short session after reading the rules.

Crucially, the environment pauses while a model generates its response. Slow inference therefore does not cause the model to miss events simply because computation takes time. The benchmark instead tests whether it can interpret changing visual evidence and select appropriate controls.

Actions execute at ten steps per simulated second. Each response can specify between one and ten actions, allowing the model to choose when to request fresh feedback. Every run lasts 30 seconds of gameplay, with immediate resets after early game termination. Each main model setting receives ten runs per game using the same seed set.

The default memory system supplies the latest frame, three recent interaction clips and three clips associated with rewards or life loss. Short textual notices identify scoring and life-loss events. This shared interface helps standardize comparisons, although the results still reflect models working within this particular setup.

Strong models still struggle with motion

Among the evaluated vision-language models, Gemini 3.1 Pro and GPT-5.4 occupy the strongest average-rank positions. Random actions generally rank near the bottom, suggesting that the benchmark rewards useful control rather than arbitrary button presses.

However, specialized reinforcement-learning agents rank first in 14 of the 16 games for which matching public checkpoints were available. These agents are trained for individual games, so the comparison is not between systems with identical preparation. It nevertheless demonstrates a substantial remaining performance gap.

To investigate the underlying skills, the researchers assembled 60 manually curated diagnostic question variants across ten games and queried each model five times per question. Across settings, perception accuracy ranges from 61% to 100%, while motion accuracy ranges from 47% to 74%. Most models perform better at identifying objects and spatial relationships than at interpreting movement.

Examples illustrate how these errors can matter. In one Breakout sequence, Gemini 3.1 Pro describes a ball moving down-left as moving down-right, then chooses repeated rightward movements. In Freeway, GPT-5.4 describes a car’s earlier position as though it were current. Higher reasoning effort does not consistently improve gameplay or diagnostic accuracy.

More history can obscure the present

Keeping the full interaction transcript, rather than selecting compact memories, usually fails to improve scores in the reported comparison across three games and three models. Such transcripts can exceed 100,000 input tokens.

One striking case involves Gemini 2.5 Flash with reasoning disabled in Breakout. During the latter half of decision turns, decisions consisting only of doing nothing account for 48.8% of turns with full-history prompting, versus 0.7% with structured memory.

Textual feedback has mixed effects. Explicit life-loss notices can help a weaker model restart play, but old reward messages can also encourage continued actions after the visual situation changes. The evidence supports careful memory design, not a blanket conclusion that history or text is harmful.

What the benchmark can establish

The authors test whether game scores distinguish the evaluated settings using comparisons against random play, contrasts between stronger and weaker settings, and statistical tests of score differences. Under their criteria, 18 games are strongly separable and two weakly separable. That supports their usefulness for these comparisons, not their validity as universal measures of intelligence.

Important boundaries remain. Eleven games are shooters, runs last only 30 seconds, and simplified two-dimensional environments do not establish real-world competence. The diagnostic set is small, while supplementary gameplay failure classifications rely on another vision-language model and descriptive samples. Scores also combine perception, memory and control rather than isolating motion reasoning alone.

AtariBench’s contribution is therefore a focused testing ground: it makes failures in updating visual understanding and converting it into action easier to observe and investigate.