An AI can match a financial answer while using the wrong history

Could an AI reproduce a historical investment analysis while quietly relying on information from the future? BuySideFlow, a benchmark introduced by Dong Bo and colleagues in a November 2026 arXiv preprint, examines that distinction. Its central finding is that matching a reference answer does not establish that the underlying workflow respected historical information boundaries.

Across nine agent configurations, the researchers audited 380 runs that passed their result checks. Of those, 193—50.8%—contained at least one confirmed point-in-time violation. That percentage describes passing runs within a shared 100-task audit, not all financial AI systems or all benchmark submissions. The work is a preprint, and its findings should be treated as provisional.

Financial records have several clocks

Point-in-time validity means using the information, historical states and record versions required for a particular decision date. A financial statement’s reporting period is not its publication date. An annual report covering December cannot automatically inform a December investment decision if it was disclosed the following March.

Historical status matters too. A company that is delisted today may have belonged in a stock screen several years ago. Applying its current status to that earlier screen can remove it incorrectly. Similarly, an industry classification that changed in July should not automatically determine where the company’s March index weight belongs.

Even choosing only records dated before a cutoff can be insufficient. When several eligible versions exist, the program must select the version required by the task. The order of operations matters: choosing the applicable classification before testing its category can produce a different selection from filtering for a desired category first.

Later outcomes are not inherently forbidden. A study can select stocks using information available at an earlier date and then measure their subsequent returns. The problem arises when those later returns influence the earlier selection or grouping.

Testing complete research workflows

BuySideFlow contains 408 Chinese-language tasks adapted from investment research practice, covering equities, funds, fixed income and macroeconomics. They run against a fixed sample of a licensed Gildata financial database containing 110 tables and 4,547 columns. Agents write SQL database queries, sometimes combined with Python for calculations, multi-step analysis or plotting.

Practitioners helped develop the questions, and specialists reviewed task statements, executable reference programs and saved outputs. Four construction-review rounds were retained. Across those rounds, 208 released tasks had at least one recorded issue, illustrating the difficulty of specifying the benchmark itself.

The evaluation separates two questions. Do the submitted results meet the task requirements? And did the executed workflow obey the temporal rules? Tables are compared programmatically; images and text use language-model judges with task-specific checklists. The workflow audit examines programs, outputs and execution evidence. AI-assisted review proposed findings, while domain experts confirmed final labels.

Each configuration contributed one retained run per task, producing 3,672 baseline records. Overall, 993 passed result evaluation, or 27.0%. Configuration-level accuracy ranged from 16.2% to 36.8%. These are research-task completion scores, not measures of investment profitability.

Where matching answers concealed mistakes

The detailed audit covered 900 runs: nine configurations on the same 100 tasks. Besides the 193 temporal violations among passing runs, auditors found violations in 345 of 520 failing runs. A violation could coexist with calculation, formatting or other errors, so the audit does not isolate temporal mistakes as the cause of failure.

One passing fund-classification submission filtered for the requested category before selecting the applicable historical version. That ordering could preserve an obsolete matching classification. The evidence established a workflow violation, but did not identify a particular obsolete classification in the submitted output.

In a separate, failing fund-manager task, the agent retrieved records for an appointment effective on 23 December but disclosed on 25 December. The research cutoff was 24 December. Its final query checked the appointment interval without enforcing disclosure availability, and included the fund.

What the evidence does—and does not—establish

An independent annotator agreed with the revised primary audit on the presence of any temporal violation in 182 of 200 submissions. Agreement varied across individual error categories. Two disputed listing conditions shifted the passing-run violation count to 188 or 198 under alternative classifications: roughly half either way.

Targeted changes to database records produced non-rounding output discrepancies or execution failures in 113 of the 193 flagged passing runs. These deliberately selected scenarios demonstrate vulnerabilities under tested conditions, not their frequency in everyday data. The study also corrected temporal-selection defects in three reference programs without changing their original snapshot answers.

Generalisability remains limited by Chinese-language tasks, one vendor’s database sample and one retained run per configuration and task. The audit did not examine hindsight leakage from model pretraining. The practical lesson is narrower than declaring these agents unusable: an answer check and a historical-validity check test different properties, and passing the first does not establish the second.