When a correct description fails to become a working curve

A curve can look plausible and still fail its job. It might connect the right endpoints but pass through an obstacle, leave the permitted region or break another rule along the way. Evaluating an AI on such a task means checking more than whether it can read a diagram or produce an equation: the equation must be executed and tested against the full geometric contract.

That is the focus of GAGR-Lab, a framework for studying how models turn spatial information into mathematical functions. This paper’s contribution is a retrospective audit of archived model calls, game episodes and mathematical checks—not a broad benchmark victory or a newly established model capability.

From a scene to an executable function

In the later tasks, models were asked to formalize scene constraints, construct a function, or judge whether a proposed function satisfied the rules. A fixed scoring process checked the outputs. In separate historical game episodes, models proposed curves that were executed in a Graphwar-derived arena. The recorded trajectories could hit a target, leave the world or collide with terrain.

The authors rechecked saved requests and responses, matched calls sharing a stage, model profile and scene, and applied exact mathematical diagnostics where the expressions allowed. They also inspected archived screenshots and trajectory data. These evidence types have different roles: a saved answer shows what a model returned; a mathematical certificate can establish whether a particular function meets a stated contract. Neither reveals the model’s private internal reasoning.

Describing rules was not the same as constructing a solution

In matched formalization and synthesis calls, the model correctly expressed the scene’s constraints but failed to construct a valid function in 30 pairs. The reverse pattern—successful construction alongside failed formalization—appeared in two. These are observations from archived requests, not proof of a universal reasoning gap. The operations had different output requirements, and scenes could recur across calls.

An exact endpoint check sheds light on some failed polynomial constructions. Of 406 failures covered by the diagnostic, 142 met both endpoint conditions. They must therefore have failed some other part of the contract. The other 264 violated at least one endpoint condition and may also have broken other rules. This check does not identify every failure’s cause, but it illustrates why checking only a curve’s start and finish is not enough.

Candidate design also affected tests of answer judgment. Across 176 positive–negative candidate pairs, 89 pairs were rejected on both sides, while 66 were classified correctly on both. Every negative candidate violated at least one endpoint condition. So a model could reject a negative using that relatively local clue, without demonstrating that it could spot a subtler problem along the full path. A stronger test would keep endpoints and other requirements intact while changing just one selected geometric property.

Auditing the information, not just the score

The researchers also checked what models actually received. All 187 requests labeled as image-based included an image and a complete symbolic description of the geometry. These were not image-only tests, so their results cannot show that models extracted the necessary geometry from pixels alone.

A separate mathematical control highlights another benchmark risk: perhaps a task does not require the instance-specific information it is meant to test. In a catalog of 72 related scenes, the authors found one fixed curve that satisfied every scene. A model could therefore succeed across that catalog without adapting its answer to each instance. This does not mean visual information is never useful. It means a test of instance-specific visual reasoning should first establish that the scenes require different answers.

Historical game traces offer concrete examples of success and failure. In one scene with nine obstacles, a sine curve left the world, while a simple straight line later hit the target. That example cautions against treating obstacle count alone as a measure of difficulty. Selected hit episodes, however, are illustrations—not a controlled comparison of models or scene types.

What remains uncertain

The study is retrospective, and its comparisons have important limits. The full archive contains 2,333 unique saved attempt identities: 1,341 main observations, 968 historical observations and 24 complex-formalization observations. Within the main observations, exploration completed 751 of 1,988 planned jobs, and a later frozen plan completed 530 of 2,016. Configurations and prompts vary, scenes are reused, and only five exploration scenes and one later-stage scene were shared across all 14 configurations. The authors report descriptive matched counts, not confirmatory significance claims based on treating dependent requests as independent samples.

Some striking figures from an earlier manuscript version were not newly verified here. The raw archive for a reported 72-game pilot and the archive for an independent analytic control remain unavailable, so those reports are excluded from the reconstructed datasets. The public accession for the new analysis package is also still awaiting author deposit. These limits affect how fully readers can independently reproduce the work.

The proposed next steps are experiments, not findings from this study. They include fixing intermediate formalizations to isolate construction, designing more diagnostic candidate pairs, controlling support-text length and testing images whose changes genuinely alter the valid answers. Until such tests run on calibrated, held-out scenes, GAGR-Lab is best read as an account of what archived behavior shows—and a guide to making future evaluations more informative. The manuscript is a preprint, not evidence of peer-reviewed consensus; several administrative declarations are still awaiting author confirmation.