Preprint Finds Hidden Evaluation Errors Can Reshuffle AI Agent Rankings

When the measuring system changes the result

In Silent Instrument Errors Reorder Agent Leaderboards, a preprint posted on November 3, 2026, Yangze Liu of Shandong University reports an audit of HAL, the Holistic Agent Leaderboard, and tau2-bench. The central argument is that an agent’s score depends not only on its model and surrounding software, but also on the infrastructure that grades, records and prices its activity.

Liu examined released code, leaderboard records and saved trajectories—the records of agents’ interactions—without running agents or calling models. The findings are the author’s reported results from those artifacts, not a comprehensive assessment of agent evaluation generally.

A failure score without a grade

The most striking comparison concerns HAL’s airline customer-service benchmark. Across nine models tested with both setups, HAL’s Generalist agent software trailed the benchmark-specific Tool-Calling setup by 13 percentage points on 41 tasks whose reference solutions queried or changed a database.

According to the audit, the two setups differed substantially in whether their episodes reached the grader. Tool-Calling episodes were graded 98% of the time; Generalist episodes, 51%. The Generalist often sent a closing message and exited without triggering the benchmark’s formal ending condition. Those episodes were stored with zero reward even though the database comparison never happened.

Replaying the agent’s own actions and grading the previously ungraded episodes raised the Generalist’s pooled accuracy from 30% to 43%, closing the pooled gap. Tool-Calling changed little.

That correction needs context. Much of the gain came from read-only tasks, where the benchmark’s existing rule accepts an unchanged database. On tasks requiring database changes, Generalist accuracy rose from 28% to 33%. Liu reports that all 23 newly passing, previously ungraded episodes in that subset contained the required write operation.

The result does not establish that general-purpose and specialized setups are equivalent. They used different step budgets and different ways of accessing airline policy. Individual model comparisons also rested on relatively few records, limiting conclusions about small remaining gaps.

Costs and logs can mislead too

A second finding concerns nested tracing records. One model request can appear both in a wrapper’s record and in a child record from the provider client. Summing both counts the same usage twice.

The audit found reported token totals exactly twice the root-span totals in 70 of 134 logged runs. The distortion was uneven: other runs were unaffected or had different ratios. That matters because inconsistent inflation changes relative cost comparisons, rather than merely rescaling every price equally.

For one airline run, the archive listed $149.98 while the root-span calculation gave $74.99. Both amounts look plausible without examining the underlying records.

The third finding concerns action provenance. To calculate rewards, the grader replayed reference actions through an environment that appended them to the same list used for agent actions. HAL then archived that combined list as the agent’s activity.

Liu identified grader-written actions in 68% of 2,334 public airline records. In 58 records, the agent made no tool call, yet the archive displayed the complete reference tool sequence. Such records could mislead researchers analyzing behavior or using trajectories as training material. Removing only trailing actions would not reliably fix them: some grader actions appeared inside the record, followed by later agent activity.

Rescoring is another measurement

The tau2-bench cases show why applying a newer evaluator to an old trajectory is not automatically a neutral update.

An announced banking grading fix moved one model from ninth place to a tie for fifth. Liu presents this as a properly disclosed correction, not a faulty fix: even a legitimate evaluator change can reorder rankings.

For airline tasks, revised criteria flipped 77 of 800 saved simulations between pass and fail, moving model scores by minus seven to plus 2.5 percentage points.

A separate telecom replay problem misidentified the operating mode of four files and rebuilt their environments without user-side tools. Replay continued despite tool warnings, producing score drops of 59–82 percentage points rather than stopping with an error.

What the audit supports—and what it does not

The study covers two leaderboards. Its expectation of similar problems elsewhere rests on shared software mechanisms, not a representative survey. HAL’s offline replay also used a stubbed user simulator, evaluating database and output criteria rather than recreating full conversations.

The paper reports that HAL maintainers reproduced its three HAL findings and added disclosures within five weeks. As of the paper’s November 3 account, archived figures remained unchanged and proposed tau2-bench fixes were still unmerged.

Liu recommends publishing graded fractions, deduplicating cost records, labeling who produced each action, and pinning evaluator and task-data versions with before-and-after rescoring results. The practical lesson is narrower than dismissing leaderboards: reproducible records make auditing possible, but do not by themselves establish that the measurement was correct.