AI agent benchmarks can lose their rankings before they run out of challenges

Two ways a leaderboard loses information

In a preprint posted on November 10, Zihan Dong, Qishi Zhan and Minxuan Hu examine released results from eight agent benchmarks and conduct a small additional experiment. They distinguish two problems that are often bundled together as benchmark saturation.

First, a pass/fail score can be too coarse to reliably distinguish nearby competitors. Second, it can stop rewarding improvement once an agent reaches a reference performance. These problems require different remedies: better tests and more observations address uncertainty, while graded scoring can preserve differences that a threshold discards.

Small gaps do not necessarily establish an order

The researchers analyzed task-level results from SWE-bench Verified, Terminal-Bench 2.0 and OSWorld-Verified. Rather than treating every difference in percentage points as a meaningful ranking, they grouped agents into statistical tiers, using comparisons that account for testing many pairs.

On SWE-bench Verified and Terminal-Bench 2.0, none of the pairs within the top ten differed significantly under their multiple-comparison procedure. Across all three benchmarks, no neighboring pair separated significantly. That does not prove the agents have identical abilities; it means these evaluations did not establish those differences reliably.

Incorrect verdicts add another complication. Using strengthened SWE-bench tests from an earlier study, the authors found that 17.7% of the median agent’s originally passing patches were rejected. Correcting those verdicts changed the leading group, but still did not reliably distinguish neighboring entries. The strengthened tests are themselves an assumed reference, not an infallible assessment.

Passing can conceal better solutions

To examine the second problem, the authors rescored existing runs using binary thresholds, partial credit capped at a reference, and graded scores that allow credit above it.

ALE-Bench provides a particularly clear example. Its official contest-performance scores separated the top ten models into three tiers. When the researchers instead counted whether each run reached the median human contestant’s performance, those models collapsed into one tier. Allowing credit above that reference restored three tiers on the same tasks.

Importantly, ALE-Bench already uses continuous scoring. This comparison demonstrates what information hypothetical pass/fail scoring would lose; it does not expose a flaw in ALE-Bench’s official choice.

Nor did graded scoring improve every benchmark. On MLE-bench, the binary measure already distinguished leading models, and the proposed objective produced weaker separation. On SWE-fficiency, partial credit below the reference helped most. The lesson is not simply that continuous numbers always make better rankings: their distribution and aggregation matter.

Measuring what the tests throw away

The team also reran four models on 13 Terminal-Bench tasks whose tests compare a numerical quantity with a threshold. Three trials per model and task produced 156 runs. The tests were modified to print the measured quantity without changing their acceptance conditions.

Three models were statistically tied on pass/fail performance, yet some successful outputs differed substantially. On a portfolio-optimization task, all their runs passed, while median speedups were 10.9-fold for one model and about 1.3-fold for the others. On a model-recovery task, median MSE ranged from 0.016 to 0.60.

These are stronger demonstrations of differences on individual tasks than of an overall model ranking. The authors explicitly describe the aggregate evidence as borderline given the small number of trials.

Separate validity, quality and cost

The proposed alternative has three parts: a hardened acceptance check, a numerical quality objective, and a separate cost meter.

For positive quantities with meaningful ratios, the authors recommend logarithmic scoring anchored at a baseline and a reference. Invalid runs score zero; valid runs receive zero at the baseline, one at the reference, and potentially more above it. A ceiling of three limits any single task’s influence. That ceiling is a design convention, not a universal constant.

Anchors would be dated and frozen for comparability. Runtime references would be remeasured on the same hardware as agents. Dollars, tokens and time would remain separate from quality, making tradeoffs visible rather than burying them in one composite score.

A narrower claim than replacing every benchmark

The proposal needs tasks with defensible numerical objectives. Many software fixes and computer-use tasks do not naturally supply one, and numerical targets can themselves be gamed. Reliable acceptance checks remain essential.

The work is a preprint, not an established peer-reviewed conclusion. Its evidence also depends on existing test audits, partly reconstructed model-release dates and a small rerun. Its practical contribution is nevertheless clear: before replacing a benchmark with harder tasks, check whether its scoring has stopped recording the improvements those tasks still reveal.