Can an AI’s mistaken memories warn of unreliable forecasts?

Testing the past before trusting the future

A language model can remember who won an award while getting the supporting details wrong. Does that imperfect recall also reveal where its predictions are likely to fail?

Prashanth Bhaskara, Shubhaankar Gupta and Seojoon Yeon investigate that question in an arXiv preprint submitted on November 7, 2026. The source record also reports acceptance as a spotlight at an ICML forecasting workshop. Their study measures associations between model responses; it does not experimentally establish that mistaken memories cause forecasting errors.

The central distinction is between remembering an outcome, remembering it accurately, and assigning a useful probability. These are related abilities, but they are not interchangeable.

Giving recalled answers a second score

The authors assembled 1,300 resolved Polymarket questions with closing dates between January 2023 and January 2025, plus a stated forecast corpus of 1,350 unresolved questions closing between September 2026 and January 2030. The markets span nine domains, including elections, economics and entertainment.

Four models entered the main recall analysis: Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5 and Gemini 3.1 Pro. Each model–market pair was queried once. Forecasts relied on the models’ internal knowledge rather than live research.

For historical questions, the researchers requested probabilities, a recalled outcome and supporting facts. Their Recall Fidelity Score assigns zero to an incorrect recalled outcome, 0.5 to a correct outcome accompanied by factual errors, and one to a correct outcome with factually consistent support. Declared failures to remember are recorded separately; the domain-level analysis counts them as zero.

Outcome matching was deterministic. When the outcome matched, Claude Haiku 4.5, equipped with web search, graded the explanation and evidence. This makes the score an automated assessment of stated recall—not a direct inspection of what a model learned during training.

Looking for nearby historical mistakes

Instead of relying only on broad subject averages, the researchers identified similar historical markets using word-weighted representations of question text and metadata. Their main local score averages recall fidelity over ten neighboring markets. Markets sharing an event identifier were excluded from one another’s neighbor sets to reduce leakage from closely related questions.

For resolved historical targets, the outcome measure was the Brier score: the squared difference between a predicted probability and the actual result, encoded as zero or one. Lower is better. A 50% prediction scores 0.25 whichever outcome occurs; a confidently wrong prediction approaches one.

Across 4,897 model–market observations, higher local recall fidelity was associated with lower historical Brier loss. The reported coefficient was −0.0652, with a p-value of 0.0048. On that fitted scale, a 0.1 increase in local recall corresponds to about 0.0065 lower Brier loss—not a 6.5-percentage-point improvement in accuracy.

The analysis compared models on the same target markets while accounting for average model differences. Negative associations persisted across neighborhood sizes from three to 50. Shuffled and otherwise less relevant neighbors did not reproduce the same relationship, according to the authors.

These are retrospective results on resolved questions, however, not a prospective demonstration of forecasting success.

Incorrect recall carried the strongest warning

The researchers also separated neighboring answers into correct, partially correct, incorrect and no-recall categories.

Relative to partially correct recall, a greater share of incorrect recalled outcomes had the strongest association with worse historical scores. Its regression coefficient was about 0.200, compared with 0.066 for no recall. Correct recall was not statistically distinguishable from the partial-recall baseline in this analysis.

The paper calls the incorrect-outcome category “confidently wrong.” Its operational definition concerns a model claiming to remember the wrong outcome; readers should not interpret the label as a separately established probability-confidence threshold.

Agreement with markets is not accuracy

For unresolved questions, the researchers could not score predictions against eventual outcomes. Instead, they measured distance from contemporaneous market prices.

Higher local recall fidelity was associated with smaller squared and absolute differences from those prices when comparing models on the same market. But the corresponding soft-target log-loss association was not statistically significant, and the domain-controlled specifications did not show a significant recall association either. The signal therefore depended on the comparison and scoring method.

Market agreement is a useful benchmark, but markets can be wrong. These results do not show that better recall beats traders or predicts eventual outcomes more accurately.

A diagnostic with unresolved limits

The sample was nonrandom: collection accepted the first qualifying markets returned by the API. Each model answered each question only once, and the automated grader was instructed to overlook some inaccuracies, including date errors measured in days rather than months or years. The fidelity score therefore does not capture every detail error.

The paper also contains reporting inconsistencies. Its open-market regression lists 1,897 market clusters, exceeding the stated 1,350-market forecast corpus; the supplied text does not reconcile those counts.

Most importantly, training-data coverage could improve both recall and probability estimates. No intervention tested whether correcting memories improves forecasts. The supported conclusion is narrower: errors about relevant past events may help identify unreliable model judgments. Turning that warning signal into better predictions remains unproven.