ModalSearcher trains AI agents to decide when a search needs images

A question can begin with words but eventually require looking at a picture. Identifying an institution might lead to a historical painting, whose visual details then supply the answer. The challenge is not simply understanding an image supplied by the user: it is recognizing when new visual evidence would help, finding it, and connecting it to other information.

In the November 4, 2026 preprint ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search, Zhihan Yin, Ruyi Li and colleagues describe a system trained for this process. They call the capability autonomous cross-modal search: choosing whether to reason directly, search text, retrieve images from a textual description, or reverse-search an existing image as information needs change.

Teaching the agent to find missing evidence

The work combines a data-generation pipeline, a trained agent and an evaluation benchmark. Its central idea is to construct questions whose supporting evidence moves between text and images, without explicitly telling the agent when to switch.

ModalSearch-Pipeline first builds a graph connecting Wikipedia entities and images. Text-to-text connections describe relationships between entities. Text-to-image connections identify relevant visual content, while image-to-text connections link depicted entities back to their Wikipedia articles.

The pipeline distinguishes images that can be retrieved reliably from a description from those whose descriptions produce substantially different plausible results. Only the former receive incoming text-to-image connections. Ambiguous images can still serve as starting inputs because the agent receives the actual picture rather than having to find it.

The researchers then sample paths through this graph and construct questions backward from the final answer. Rewriting removes entity names and image targets that would reveal shortcuts, while retaining clues needed to resolve the question. Importantly, the authors acknowledge that making a visual dependency implicit does not eliminate every possible textual route to an answer.

The graph is a construction aid, not the agent’s search database. At inference time, the agent searches the live web without receiving the reference path.

From examples to live search

ModalSearcher-8B starts from Qwen3-VL-8B. Supervised fine-tuning uses approximately 16,000 examples, combining pipeline-generated questions with training examples from FVQA and LiveVQA. GPT-5.4 generates expert search trajectories, with incorrect answers, failed tool calls and insufficiently supported trajectories discarded.

Reinforcement learning then uses 5,871 additional generated examples to optimize multi-round search decisions. Rewards emphasize final-answer correctness, with a smaller component for valid interaction formatting.

The runtime environment offers text search, text-to-image search, reverse-image search and a tool for reading webpages, PDFs or images. The agent repeatedly examines evidence and decides whether to answer or take another action.

One reported inference example traces clues about a Japanese university to Chuo University, then to Middle Temple in London and a portrait of Charles I with his riding master. After retrieving and inspecting the painting, the agent answers that the horse’s left foreleg is raised. The example illustrates the intended movement from textual identification to visual inspection.

What the evaluations show

Across five multimodal knowledge-intensive question-answering benchmarks, the authors report an average accuracy of 57.0% and a 16.2-percentage-point improvement over Qwen3-VL-8B using the same agentic workflow.

On the independently constructed VisBrowse and BrowseComp-V3 benchmarks, ModalSearcher reaches 33.4% and 21.3% accuracy. Those results exceed the base model by 19.8 and 15.8 percentage points, respectively. They support transfer beyond questions generated by the researchers’ own pipeline, although accuracy remains far from complete.

The team also introduces ModalSearch-Bench: 300 expert-screened questions drawn from a separate graph, with nodes, paths and questions held out from training. It contains 100 Input-only questions and 200 Search-required questions. ModalSearcher scores 45.8% overall. Because this benchmark shares the training data’s construction procedure, the authors distinguish its controlled evaluation from the external tests.

Images help, but the evidence has boundaries

Removing text-to-image search lowers ModalSearcher’s accuracy by 9.2 percentage points on VisBrowse and 8.5 points on ModalSearch-Bench. These interventions support a performance contribution from visual-search access, rather than merely showing that the agent invokes image tools more often.

Other measurements require narrower interpretation. Reported visual-target alignment and coverage track what the agent searches for, not whether returned images are correct. Likewise, repeated GPT-4o judging measures consistency, not the correctness of the judge’s decisions.

The preprint therefore supports a specific conclusion: training on interleaved evidence paths can improve search decisions and benchmark answers. It does not establish that every designated Search-required question has an exclusively visual solution, or that autonomous visual research is solved.