FondueQA tests how AI fills gaps in retrieved evidence

Language models paired with search systems have two sources of information: retrieved documents and facts learned during training. When retrieval misses a crucial document, a model may still answer correctly. But does it recover the missing fact, connect the remaining evidence, or recognize a question it has encountered before?

A new preprint by Laurin Schmid and colleagues introduces FondueQA to investigate these possibilities more systematically. Its central finding is that recalling information and using it successfully within a larger question are not the same capability.

Making missing evidence measurable

The study focuses on questions that require combining multiple facts. Consider the paper’s example: asking for the nationality of the director of Titanic requires identifying James Cameron as the director, then identifying his nationality. The first fact provides a bridge to the person; the second supplies the answer.

Existing benchmarks can make evidence-removal experiments difficult to interpret. A supposedly removed fact might also appear in another document, an irrelevant passage, or the question itself. Older public questions might also have appeared in model training data.

FondueQA attempts to control the evidence problem by assigning each required fact to one designated document and checking that it appears nowhere else in the supplied context. Removing that document should therefore remove that fact from the material given to the model.

The researchers generated questions from Wikidata and used articles from an October 2017 English Wikipedia snapshot as evidence. They combined rule-based checks with semantic assessments by GPT-OSS-120B. Unsuitable examples were repaired and checked again, or discarded, without editing the source articles.

The published release contains 13,063 questions, each with 20 context documents. Questions include factual chains, comparisons, and choices where either, both, or neither of two candidates may qualify.

The connection can matter more than the answer

The researchers evaluated models from the Qwen, Gemma, GPT-OSS, and DeepSeek families, using standard and reasoning modes where supported. Models were explicitly allowed to use their internal knowledge when documents lacked information.

Experiments compared complete evidence, selected missing documents, contexts containing no supporting documents, and questions presented without context. The main measure was exact-match accuracy, with normalization and accepted alternative answer forms.

On FondueQA, retaining the bridge fact often helped more than retaining the answer-bearing fact, particularly for smaller models. Knowing which director the question concerns could be more useful than seeing a nationality attached to a person whose connection to the film was missing.

This pattern persisted when the remaining supporting document occupied the same position in both conditions. However, it was not universal. Adapted versions of HotpotQA and 2WikiMultiHopQA generally favored answer-bearing evidence, while the adapted MINTQA benchmark favored bridge evidence. The study therefore does not establish a single ranking of which document matters most.

Familiar facts are easier to replace

Removing evidence about more popular entities generally caused smaller accuracy losses. The researchers estimated popularity using Wikipedia pageviews, finding a similar broad pattern across the evaluated benchmarks.

That measure needs care: pageviews indicate public attention, not what a particular model actually encountered during training. The results support an association between popularity and successful compensation, rather than directly measuring a model’s stored knowledge.

Separate questions about individual missing facts revealed another mismatch. Models sometimes recalled a fact on its own but failed to use it in the larger question. Conversely, they sometimes answered the larger question correctly after failing the isolated recall test. Remaining context may help identify relevant entities or narrow the possible answers, but the experiments do not establish the internal mechanism.

Plausible alternatives can redirect answers

A further experiment removed the bridge document while retaining the true answer-bearing document. The researchers then replaced distractors with documents about plausible alternative entities, keeping the total at 20 documents.

With randomly selected competitors, some reasoning-mode configurations lost up to about 15 percentage points of exact-match accuracy. Effects varied across models. Competitors supporting the correct final answer could improve scores, while competitors consistently supporting another answer could reduce them substantially.

These results suggest that retrieval quality involves more than including relevant facts: the surrounding documents can influence which connection a model makes.

The findings remain bounded by the benchmark’s design. Automated semantic checks can make mistakes, historical Wikipedia evidence limits the material tested, and filtering changes the composition of adapted benchmarks. Newly constructed questions also do not prove an absence of training overlap. As a preprint, the work should be read as research evidence rather than a settled account of model reasoning. Its practical lesson is narrower: successful factual recall alone does not guarantee reliable answers when evidence is incomplete or ambiguous.