AI diagram errors shift from counting to seeing when code takes over

A multimodal AI can correctly identify a diagram’s connections and still give the wrong numerical answer. Conversely, a program can calculate perfectly from a graph that the AI has misread. Yin Liu’s arXiv preprint, submitted on November 6, 2026, investigates both failures. This report draws on the supplied full paper and appendices, not just its abstract. The supplied material does not establish peer-review status.

Different questions, the same picture

The study introduces “complexity twins”: counting questions about the same drawing that have different mathematical difficulty classifications. One example compares spanning trees—ways to connect every vertex without forming a loop—with simple paths, which never revisit a vertex. Spanning-tree counts can be calculated using a matrix determinant. Counting simple paths is #P-complete on the relevant graph families, a classification associated with difficult exact-counting problems.

Crucially, the theoretically harder question need not have the larger answer. One illustrated graph has 324,268 spanning trees but only 200 simple paths between its designated endpoints. Holding the picture fixed helps separate visual difficulty from computational difficulty; matching answer sizes provides a further control.

The benchmark covers eight tasks involving grid graphs, black-and-white cell images and drawings of circuits that traverse every edge. Its core contains 392 image–task combinations, expanded to 921 combinations. The author reports that an independent implementation reproduced all reference answers on 344 core, expansion and extra-large instances.

Reading correctly is not enough

The experiments compare image-based answers with answers from exact text descriptions. Image prompts require a transcription before counting, allowing the author to score the extracted structure separately. Other conditions supply algorithm hints or ask models to generate Python programs, executed with a 60-second limit.

The main configuration was Gemini 3.8 Flash with low thinking enabled. Of its 144 wrong answers on the core visual benchmark, 115—about 80%—came with an exact transcription. The remaining 29 also failed when given the correct structure as text, so those cases cannot cleanly identify which step caused the original failure.

A separate experiment obtained one task-independent transcription per image before asking counting questions. It recovered 378 of 384 images exactly, yet downstream counting still produced hundreds of errors. This supports a computational bottleneck for that model on these benchmark-sized drawings, rather than showing that perception never matters.

Answer magnitude was strongly associated with failure. In a regression covering 721 grid and circuit tasks, each tenfold increase in the true count divided the main model’s odds of an exact visual answer by 2.38. That describes odds, not a fixed percentage-point decline in accuracy.

Mathematical class contributed a smaller, less stable association. On 80 specially generated twins with closely matched answer sizes, the lower-complexity task won about 60% of comparisons where only one answer was correct, pooled across eight configurations. Evidence was borderline for visual inputs and stronger for text inputs. More thinking did not preferentially improve the lower-complexity member.

Code moves the obstacle back to vision

On the core benchmark, generating executable code raised the main model’s accuracy from about 63% to 93%. On 96 fresh grids ranging from 7×7 to 10×10, its programs answered 216 of 240 tasks correctly from exact text, compared with 66 from pictures. Chain-of-thought answers without executed code solved none of those 240 tasks, whether the input was visual or textual.

Large-image reading frequently missed horizontal connections. A repair pipeline read each drawing upright and again after rotation, then checked disagreements using small crops showing the relevant vertex pair. It accepted agreements and sent the resulting graph to code, without consulting ground truth.

On the 96 fresh grids, both 90-degree and 180-degree second views produced 86 exact transcriptions, compared with 57 when the upright reading was simply repeated with the same verification procedure. The equal results for both rotations matter: the benefit cannot be attributed specifically to turning horizontal lines vertical. Differently rendered views exposed errors that repeated upright readings often shared.

What remains uncertain

The findings are model-dependent. Self-hosted Gemma and Qwen3-VL configurations often struggled to transcribe even benchmark-sized diagrams. Larger thinking budgets helped many counting conditions, but part of that improvement reflected fewer responses ending before an answer was produced.

All images were generated from known structures, including realistically styled circuit boards and transit maps; natural photographs with occlusion were not tested. Only four of eight registered criteria passed for the main model in the realistic-style extension. Answer magnitude also covaries with instance size, making its association observational rather than proof of a particular internal algorithm. Finally, complexity classifications concern scaling: bounded-width grids can admit efficient exact methods for both twins. The study therefore diagnoses specific failure regimes, not a universal limit on AI vision or counting.