SARBLOH Turns Game Grids Into Spatial Clues for AI Agents

Why seeing the board matters

ARC-AGI-3 asks agents to learn unfamiliar visual games without instructions. They must discover what their actions do, identify objectives and complete increasingly complex levels. Observations are grids of categorical colors, with dimensions up to 64 by 64 cells.

The benchmark makes wasted moves expensive. Its per-level action-efficiency score depends on the squared ratio between a human action baseline and the agent’s action count, subject to a cap. Taking twice as many actions as the human baseline earns only one-quarter of the available baseline credit. Internal computation does not add to that action count, although the offline competition imposes a nine-hour execution limit.

Banwait argues that agents can waste this budget because they misunderstand spatial structure, not simply because they cannot plan. SARBLOH addresses that proposed bottleneck by calculating a structured description of the board outside the reasoning model.

Separate measurements from guesses

The system’s central principle is the Measured–Guessed Rule: distinguish geometric relationships computed from the grid from hypotheses about what objects do.

A deterministic perception engine groups cells into objects and tracks their identities across frames. It calculates relationships such as alignment, enclosure and shape matches under rotation, reflection or integer scaling. It also computes reachability over a detected movement lattice.

Object roles are handled differently. A compact, multicolored shape might be the player; a long bar near the screen edge might represent remaining moves. Those labels come from fixed heuristics rather than learned parameters, and the engine presents them as guesses with proposed verification actions. The model receives a schematic map and a text briefing that separates the two categories.

This distinction matters because matching a shape or measuring its position does not establish its gameplay function. The architecture is intended to keep plausible interpretations from becoming unquestioned rules.

What the experiments show

In a perception lab, Banwait compared nine presentation formats, including a legacy baseline, on the opening frames of two development games. A 28-point rubric assessed recognition of features including the controllable entity, goal, mechanical links, gauges and winning plan.

The legacy text encoder scored 3 out of 28 while using approximately 3,500 text tokens per frame. The combined Fused Briefing scored 25 out of 28 at approximately 370 tokens, a reported 89.4% reduction in text-token use. Scoring was non-blind against an answer key written before the presentations were evaluated; these are scene-comprehension results, not full-game completion scores.

Checks on 23 additional games reinforced the need for caution about roles. Grid reconstruction achieved exact cell recovery across all 23. By contrast, candidate player identification succeeded in 6 of 14 keyboard games, remaining-move gauge detection in 10 of 20 games, and click-handle predictions in 7 of 29 cases. A single-frame predictive presentation was fully correct in only 1 of 16 games.

A separate diagnostic on one game’s opening level found that a frontier model exhausted its budget after 200–300 actions with a raw integer grid, while a rendered image required 12–24 actions. Adding structural explanations and then executable search reduced the count further. However, this was a single exploratory trial, not a repeated estimate of typical performance.

Code turns interpretations into actions

SARBLOH lets the reasoning model write Python to model a game, search for a solution and submit a sequence of moves within the same execution environment. An optional predictive model can compare expected and observed states.

The detailed execution contract makes an important distinction: a prediction mismatch is recorded but does not itself stop the sequence. Execution stops under specified conditions, including level completion, game termination or repeated unchanged moves. Some additional guards are configurable.

Stronger scores, qualified comparisons

The paper reports a hidden Kaggle evaluation score of 16.53 with the full perception engine active, approximately 13.7 times the first Milestone winner’s reported 1.21. This competition metric should not be confused with public-game efficiency percentages.

On the public set, Run A completed 143 of 183 levels in 8.2 hours with a mean score of 59.4%. Crucially, its briefing and schematic map were disabled. Run B enabled them and completed 138 levels in 5.4 hours with a mean score of 63.5%, but also changed guardrails, stuck-level review and context handling. With one run per configuration and several differences, the comparison does not isolate perception’s causal effect.

The preprint therefore offers evidence for structured spatial representations, not proof that planning is generally secondary. Remaining limitations include unreliable initial role guesses and the absence of a unified belief distribution updated across multiple steps. The practical proposal is narrower: compute useful geometry explicitly, and keep gameplay interpretations open to testing.