Financial AI benchmark shows how fact-checking can change the leaderboard

An answer that looks like strong financial analysis can lose its advantage once its factual claims are checked. In AgenticFinBench, a preprint from NashNova Research posted to arXiv on November 3, 2026, the configuration leading under rubric-only scoring dropped to fourth after an evidence audit. Its average score fell by 7.58 points.

The study introduces a benchmark for financial agents: systems that combine a language model with software for using tools, retrieving information and producing answers. Its central distinction is between answering a request well, supporting statements with evidence and getting those statements right. These overlap, but the researchers argue that they should not be treated as interchangeable.

Financial questions without a single answer

AgenticFinBench contains 230 questions drawn from real-world requests, including 189 in Chinese and 41 in English. They cover eight categories, ranging from financial concepts and calculations to company research, event analysis and investment planning.

The researchers assign four complexity levels, from retrieving a single fact to designing an analysis or investment plan. Almost two-thirds of the questions fall into the two higher levels.

The collection is curated rather than representative of all production traffic. The team filtered requests, removed personal information and used expert review. Different collection batches used different sampling methods, including purposive selection for the English batch.

Because open-ended financial questions can have several defensible answers, the benchmark does not score against a reference answer. Instead, each question has a rubric covering task completion, reasoning, evidence and accuracy, and scope and applicability. Positive criteria earn points; specified errors trigger penalties and, for some serious failures, score caps.

Unsupported does not automatically mean false

The evaluation splits factual statements into individual claims and checks them in two stages. First, an auditor examines whether the agent’s execution trace supports each claim. Tool results, retrieved documents and calculations with recorded inputs count as evidence. The model’s own reasoning does not.

Second, the auditor assesses factual correctness, reusing reliable trace material and seeking additional evidence when necessary. A claim is refuted only when evidence explicitly contradicts it. Insufficient evidence leaves it unverified.

Under the paper’s deliberately narrow definition, a hallucination must both lack trace support and be refuted by evidence. A correct definition supplied without retrieved support is therefore “verified but ungrounded,” not hallucinated.

Conversely, an agent can repeat an error from a retrieved source and remain classified as grounded. Such confirmed errors can still reduce its quality score. The reported hallucination rate is consequently not a complete factual-error rate.

The scoring procedure also groups errors by root cause and credits points already lost through related positive criteria, avoiding duplicate charges for the same mistake.

Auditing changes the comparison

The researchers evaluated 18 configurations, producing 4,140 responses. Each configuration combines a model, a harness—the software directing its tools and workflow—and a reasoning setting.

DeepSeek V4.1 Flash on Pi fell from first under rubric-only scoring to fourth after auditing. GPT-5.6 Sol on Codex CLI lost 0.27 points and moved from fifth to first, reaching a mean quality score of 89.20. However, the top three audited scores were within 0.6 points, and the paper cautions that adjacent ranks need not differ significantly.

Across configurations, just 0.03%–2.38% of extracted claims met the hallucination definition. Yet 0.9%–47.0% of responses contained at least one such claim. The distinction matters: a low proportion of erroneous claims can coexist with a much larger share of affected answers.

Harness differences also mattered. In a complete comparison of three models across three harnesses, harness differences accounted for 28.6% of question-centered variation in verified-but-ungrounded claim rates, versus 8.8% for models. That result concerns evidence grounding, not overall quality; for quality scores, the model component was larger. Instructions, tools and execution settings differed between configurations.

Harder analysis remains harder

Averaged across configurations, quality fell from 89.44 on direct-retrieval questions to 79.18 on the most complex tasks. Exploratory analysis identified explaining financial impacts and comparing or combining information as the largest sources of unmet reasoning requirements. That classification was not independently double-coded by humans.

Expert checks provided support for the evaluation approach, without making it infallible. Audited scores matched adjudicated expert preferences on 73.3% of random answer pairs, compared with 67.8% for baseline scores.

These are preprint findings, not peer-reviewed conclusions. The selected questions and rubrics shape the results, and observed links between financial-data use and stronger grounding do not establish causation. The benchmark’s contribution is a more explicit separation of analytical quality, recorded evidence and factual correctness—not proof that a high-scoring agent is ready to make investment decisions.