When Language Models Follow a False Lead

Language-model hallucinations are often discussed as if they have one main cause: the model’s design, its training data, or the way it generates text. This study asks a more specific question: when a model encounters a false statement, does its architecture affect whether that statement shows up in the answer? And when an answer starts with a mistake, does changing that first mistake reduce later errors?

The authors tested these questions through two experiments that changed prompts or model components and compared the resulting answers. Their evidence is behavioral: it shows what the tested models did, not a complete account of the internal processes that produced those outputs.

Testing responses to misleading context

In the first experiment, the researchers compared Pythia and Mamba language models in five approximately size-matched pairs, from around 130 million to 2.8 billion parameters. The models were trained on the same corpus and used the same tokenizer, but differed in their main context-mixing approach: Pythia uses attention, while Mamba uses a state-space mixer. Four additional attention-based models provided further comparisons.

The researchers gave models questions about 111 facts, sometimes with a planted false statement in a passage and sometimes with the true statement. They varied where the statement appeared and how long the passage was. Controls tested whether a model could read a supplied fact it did not already know, and whether the models being compared already knew the same facts.

On a separate free-recall test, the 2.8-billion-parameter Pythia and Mamba models each answered 3 of 200 SimpleQA questions correctly. That result does not show that attention never affects recall: the models were near the bottom of the test’s performance range. But it provides no evidence that replacing attention improved their performance on those questions.

The context test produced a different pattern. From the 410-million-parameter pair upward, attention-based models repeated planted falsehoods more often than their matched Mamba counterparts. For the largest pair, Pythia repeated the false statement on 650 of 666 conflict items, compared with 566 for Mamba. The difference remained when researchers looked only at facts both models had answered correctly without context. Mamba models could also use some planted facts they did not know beforehand, which suggests that their lower rate of repeating false statements was not simply an inability to read the supplied line.

These results suggest that the choice of context mixer is associated with how strongly a model follows a misleading passage. They do not establish that attention alone caused the difference: the model families differ in other ways too. The authors also note that their reading control for the largest Mamba model involved only eight previously unknown facts.

What happens after an answer starts wrong?

In the second experiment, researchers examined 111 erroneous answers from three instruction-tuned models. Each answer began with a wrong fact and continued with further claims. The researchers replaced the first wrong fact either with the correct answer or with a different plausible but wrong answer, then had the model continue. They also asked the original question separately, without the earlier answer.

After the correct fact was inserted, the share of answers containing at least one downstream error fell from 92.5% to 48.1%. But inserting a different wrong fact reduced errors about as much: the difference between the two substitution conditions was not statistically significant. Asked the question on its own, the models gave the correct answer on only 2 of 111 items.

So changing an early statement altered what came next, but this experiment does not show that the model knew the truth and was trapped by its earlier mistake. The questions were selected because the models had answered them incorrectly, often on relatively obscure facts. The authors say that any replacement may disrupt the chain of elaboration, or that later claims may be separate fabrications. Their experiment cannot distinguish those explanations.

A narrower claim—and open questions

Taken together, the findings support a limited version of the structural account: the way a model mixes context can affect whether misleading input appears in its answer, and changing an early answer can affect what follows. On the long-tail questions tested, however, the first error often reflected missing knowledge. The study does not show that architecture matters more than data; it did not vary the models’ pretraining data to test that comparison directly. Nor did it directly test whether the pretraining objective’s lack of an explicit truth term causes hallucinations.

The experiments use different kinds of models and tasks. The context passages were synthetic and in English, and answers were partly scored by automatic string matching. The second experiment relied on a small set of qualifying errors, with 72 of 111 coming from one model. Some study annotators were authors, though scoring was blind to experimental condition and agreement was reported as high. The authors also identify a test of how fact frequency affects errors as future work.

This work is an arXiv preprint, not a peer-reviewed publication. Its results are useful as controlled evidence about these particular models and prompts—not a final verdict on why language models hallucinate generally, or how every deployed system will behave. The authors report releasing raw outputs and analysis code, but further tests, including on instruction-tuned models with matched architectures, would help clarify how broadly the findings apply.