When AI Uses a Tool, Did the Tool Actually Help?

A tool call is not proof of a tool’s value

The paper opens with an example: asked what material a glove in a street-food image is made of, an AI writes code that reloads and displays the original image, without cropping or otherwise processing it. The authors use this to illustrate that a tool call does not, by itself, show that the tool helped. In their example, a model without tools also answered correctly.

A tool might provide new information, confirm what the model already inferred, or fail to help. In “With Tools or Because of Tools? Tracing Capability Gains in Multimodal Agents,” Jiawei Guo and colleagues ask whether tool access expands the set of problems an agent can solve—not simply whether the agent calls a tool or earns a high benchmark score.

Comparing agents and non-tool settings

The researchers study Thyme and DeepEyesV2, two multimodal systems that can use code execution during reasoning. Their benchmarks cover real-world image understanding, optical character recognition (OCR), charts and mathematical reasoning.

For each system, the authors compare ordinary tool-enabled operation with a Tool-Free version: the same released agent prompted not to call tools at inference time. They also compare a separately trained Pure-Text Reasoner, initialized from Qwen2.5-VL-7B and trained on processed data from the DeepEyesV2 source pool without tool-calling trajectories. Its setup includes re-annotated training examples and a filtered subset of reinforcement-learning data; it does not exactly reproduce DeepEyesV2’s training conditions. The authors therefore treat it as a diagnostic reference, not a strict matched control or an upper bound. Their attribution analysis also uses the Tool-Free Agent, so it does not depend solely on the Pure-Text Reasoner.

The team examines whether tool-enabled successes were also achieved in non-tool settings, and divides outcomes on tool-called samples into shared successes, shared errors, successful corrections and cases where tool use coincided with an error. For a separate process analysis, a language-model judge assesses what tool outputs contributed; researchers manually audited a random 40% of the judged trajectories. The authors describe these labels as diagnostic, not ground-truth causal attribution.

Finally, they alter recorded tool-use sequences. In one test, the model’s tool-call message remains but the returned result is removed; in another, the result remains but the call message is removed. These interventions probe which parts of the sequence are associated with the agents’ behavior.

Most tool-enabled successes were not unique

Across the tested benchmarks, tool access did not produce a consistent overall accuracy advantage. Tool-Free versions often stayed close to their tool-enabled counterparts, and the Pure-Text Reasoner matched or exceeded them on some benchmarks. Tool use also did not reliably reduce the number of tokens generated by the model, the study’s measure of generation cost.

The clearest result concerns overlap in solved problems. Of the problems DeepEyesV2 answered correctly with tools, 93.2% were also answered correctly by at least one non-tool setting. For Thyme, the figure was 96.0%. Thus, only a small share of tool-enabled successes was missed by both non-tool references. Some individual benchmarks had larger tool-only portions, so the overall pattern does not mean tools never helped.

On tool-called samples, successful corrections—where a non-tool reference was wrong but the tool-enabled agent was right—were below 10% overall for both agents. The authors report the same general pattern when using the Pure-Text Reasoner as the reference, without specifying that the below-10% figure applies separately to each reference. These rates are not calculated over all benchmark examples.

The process analysis adds another perspective: among calls the judge classified as supplying new, correct information that was used, the Tool-Free Agent had already answered correctly on 59.7% of DeepEyesV2’s and 79.4% of Thyme’s examples. Even apparently informative calls often were not necessary for correctness. These judgments do not prove what caused an answer.

The altered-trace tests differed by system. DeepEyesV2’s behavior was more closely preserved when the tool-call format remained but its result was removed; Thyme’s was more closely preserved when the result remained but the call message was removed. Neither pattern showed a consistent benchmark advantage for the full tool-use loop. The authors interpret this as evidence that learning a tool-calling pattern is not the same as reliably learning when tool outputs expand what can be solved.

What the study can—and cannot—tell us

The practical caution is straightforward: a correct answer accompanied by a tool trace is not enough to credit the tool. Evaluations should test whether tool access changes which examples a system solves and whether outputs supply useful new information.

The conclusion is bounded. The study covers two agents and selected benchmarks, not every kind of model, tool or real-world task. The Tool-Free Agent is an inference-time diagnostic, while the Pure-Text Reasoner’s different training setup limits direct comparison. Process attribution relies partly on a model judge, despite the manual audit, and the authors do not treat it as definitive causal evidence. The comparisons reveal patterns of overlap and behavior; they do not establish that tools can never help or provide a universal estimate of their benefit.

This article is based on an arXiv preprint, which may not have undergone peer review. Its results are best read as a focused evaluation of two systems, not a verdict on tool use in AI generally.