Reading a Chart Accurately Did Not Guarantee Better AI Decisions

Two tests of visual reasoning

The preprint by Keyty Loza of Lamar University and Shayan Monadjemi of Oak Ridge National Laboratory examines how vision-language models interpret probabilities and how an agent uses visual information to choose an action. The manuscript identifies itself as presented at the Workshop on GenAI, Agents, and the Future of VIS at IEEE VIS 2026.

The researchers used simplified lottery scenarios rather than complex scientific figures. Winning probabilities ranged from 5% to 100% in five-percentage-point increments. Five visual formats represented those probabilities: icon arrays, pie charts, circle charts, triangle charts and bar charts. The potential prize stayed at $100, while the decision experiment varied the entry cost from $10 to $100 in $10 increments. The authors report repeating each experimental condition five times.

The work separates a perceptual question—what probability does a model read?—from a behavioral question—what does an agent decide to do?

Which models read the probabilities best?

In the first study, four models received a visualization and were asked to return only a numerical probability. The models were Gemma-4, Granite3.2-Vision, GPT-4o and Qwen3-VL:8b, using the names reported in the paper.

GPT-4o had the lowest overall mean absolute error, at 0.083 on the zero-to-one probability scale. Qwen3-VL:8b followed at 0.118, Gemma-4 at 0.128 and Granite3.2-Vision at 0.203. For perspective, an error of 0.083 corresponds to 8.3 percentage points of probability.

Visual format also mattered. The authors report particularly accurate estimates for pie charts and a mean absolute error of 0.078 for bar charts across the tested models. Icon arrays tended to produce overestimates, while circle and triangle charts tended to produce underestimates. Triangle charts showed the strongest underestimation across the tested models.

The direction of error differed between models, too. GPT-4o and Qwen3-VL:8b generally underestimated probabilities, whereas Gemma-4 and Granite3.2-Vision tended toward overestimation. These results describe performance on this controlled collection of images, not a general ranking for every visual task.

From estimating odds to choosing an action

The second study tested only GPT-4o. The researchers selected it because it could process images and call tools, and because they had access to it as a scalable service. Through a Model Context Protocol server, the agent could choose either to bet or not to bet. A text-only condition provided a comparison with the visual formats.

The benchmark defined the preferred action through expected net return: the probability of winning multiplied by the $100 prize, minus the cost of playing. An 80% chance of winning with a $60 entry cost therefore gives an expected net return of $20, making betting the favorable choice under that rule.

Here, “rational” has a narrow meaning: agreement with the experiment’s expected-return criterion. It does not establish that the agent exercised broadly sound judgment or that the same criterion captures every person’s financial preferences.

The authors found better alignment with this rule for circle charts, icon arrays and text than for the other tested formats. That contrasts with the strong probability-estimation performance of pie and bar charts. For those two formats, they also report an unexpected decline in decision alignment at higher positive expected returns.

The reported decision matrix shows an asymmetry. When the benchmark favored not betting, the agent chose not to bet in 84.5% of cases. When it favored betting, the agent bet in 74.1% of cases. These are measures of agreement with a decision rule, not observed gambling profits.

What the study leaves unresolved

The results suggest that accurate probability estimation alone is insufficient to characterize an agent’s decision behavior. They do not isolate why particular formats produced different choices, or establish that perception errors explain every decision error.

The authors emphasize the study’s limited models and visualization types. Only one model underwent the action-taking evaluation, and the images represented known probabilities in a simplified setting. Real scientific visualizations can include multiple variables, annotations and contextual information absent here.

Testing additional models, richer visualizations and different monetary stakes remains future work. The practical lesson from this preprint is therefore bounded: evaluating an agent’s chart reading and evaluating its resulting actions are distinct tasks, and success on one should not be treated as proof of success on the other.