Measuring how far an AI’s financial memory reaches

Why financial memory matters

A historical backtest asks how a strategy would have performed using information available at the time. A language model can undermine that premise even when its supplied documents respect the date: its internal parameters—the numerical settings learned during training—may retain subsequent earnings results or market movements. The paper calls this parametric look-ahead bias.

Siyuan Li and colleagues introduce FinCutBench to measure the calendar-time reach of that knowledge. Their study was posted on November 8, 2026, as an arXiv preprint. Its findings are preliminary, and the benchmark measures behavior rather than directly inspecting training records.

A monthly financial knowledge test

The dataset contains 9,000 four-option questions covering January 2023 through September 2026: 200 questions per month, split equally between Chinese questions about Chinese markets and English questions about U.S. markets. Four categories cover macroeconomics and policy, sectors and supply chains, market-wide price paths, and individual asset movements.

Questions come from economic indicators, market data and securities research reports. Numerical answers undergo independent calculation checks. Report-based questions are generated and reviewed by AI agents in separate contexts. Three human experts also assessed a selected sample: all 126 final report-based questions received majority approval. That supports the reviewed sample’s usability, not the perfection of the entire question bank.

Facts are assigned to their approximate public-availability month using source-specific rules. These dates matter: an economic statistic describing September might only become available in October. The authors plan monthly additions through the project repository.

Each tested model answers each question three times, without tools, supporting evidence or previous questions in its context. Averaged correctness produces a monthly knowledge profile. Random guessing among four options would average 25%, although reasoning from the options can also contribute to correct answers.

What the boundary means

The researchers first test whether earlier accuracy exceeds both chance and later accuracy, correcting for the multiple comparisons made while searching. They then fit three phases: a high plateau, a linear decline and a lower plateau. Transition Start marks the end of the high phase; the FinCut Boundary is the first month at the fitted lower level.

Checks test whether these dates remain similar when the required lower plateau is lengthened. If necessary, the observation window expands backward. A boundary is not reported when the evidence or fitting checks are insufficient.

Crucially, this is not the last month about which a model knows anything. The lower plateau can remain above chance, and individual later facts may still be remembered.

The procedure is repeated on 1,000 resampled question sets. The authors report how often a boundary can be located and the middle 90% of dates among successful repetitions. Recovering a date frequently does not necessarily mean locating it precisely.

Different models, different temporal profiles

Nine of the ten main models yielded boundaries. The reported estimates include June 2026 for GPT-6-Astra and DeepSeek-V4.1-Flash-1017, and March 2026 for Qwen3.8-27B and Gemini-3.8-Flash.

Precision varied substantially. Qwen3.5-122B-A10B had a December 2025 point estimate, but the middle 90% of dates among successful resampling runs extended from February 2025 to January 2026. Qwen3.5-4B showed no significant decline across its observation range; that does not establish unlimited knowledge. Information categories also differed within models, limiting what one overall date can describe.

Remembering outcomes is not the same as forecasting

The authors fixed the independently measured dates before comparing downstream tasks. Four models ranked assets and recalled quarterly earnings, with changes measured relative to Qwen2.5-72B answering matching questions.

Across the four models, relative U.S. ranking performance fell by 0.331 in rank correlation, a measure of agreement with realized returns. The unadjusted one-sided p value was 0.0049. The Chinese-market decline was smaller and not statistically significant. Earnings recall showed significant control-adjusted declines for GPT-5.6-Sol and GPT-5.6-Luna.

Forecasting was less conclusive. All eight comparisons of changes in the main earnings-forecast error measure relative to the control had unadjusted p values above 0.10. This does not prove no effect exists. Forecast samples were grouped by when results were disclosed, not a shared post-boundary forecasting-decision window.

Limited assets and months, approximate disclosure dates, question-generation biases and changing market difficulty constrain interpretation. Period differences do not isolate memory as their cause. FinCutBench therefore offers an empirical reference for choosing evaluation windows and interpreting results—not a contamination detector for every prediction or proof of genuine forecasting skill.