Messier gathers agent benchmark results—and shows how scoring shapes the picture

Evaluating an AI agent is not quite like marking a multiple-choice test. An agent may navigate a website, modify code, or carry out a sequence of steps in a simulated workplace. A “scaffold” translates the model’s outputs into actions, while one or more verifiers judge the result of an attempt. Benchmarks differ in all these ingredients—and in how they turn verifier results into a score for each trial, or rollout. That makes comparisons across benchmarks difficult, even when results are public.

The authors of Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation assemble scattered records into one structured resource. Their central aim is not to declare a universal winner, but to make it possible to inspect performance at a finer level than a single leaderboard score.

Building a shared record

Messier combines public evaluation data with new runs on six benchmarks where detailed results were less available. The paper reports a corpus covering 30 benchmarks, 745 model-and-scaffold combinations (which the authors call agents), 11,891 tasks, 71,969 verifiers, and 923,158 records. The new evaluations used five models with the same OpenHands scaffold; the authors report that these runs cost about $10,000 in total.

To bring different sources together, the team mapped their records into a common format linking agents, tasks, environments, verifiers, and scoring rules. They normalized model and scaffold names, ran automated checks for issues such as duplicate task identifiers, and manually checked selected entries. Tasks were also assigned occupational and industry categories using a process that included multiple language-model classifiers and adjudication when they disagreed.

A key distinction in this format is between a verifier result and the scored outcome of a trial. A task can be attempted more than once, and each trial may have several checks—for example, whether separate parts of a workflow were completed. Under an all-pass rule, every check for that trial must pass for its scored outcome to be successful. Other rules might give partial credit or count a trial as successful once a threshold of checks passes. The task is the problem being attempted; the trial result summarizes one attempt under a specified scoring rule.

Uneven performance—and a scoring wrinkle

The paper groups benchmarks into programming, research and reasoning, enterprise workflows, graphical-interface navigation, and function calling. Its frontier analysis takes the best observed result for each task among eligible agents at a given point in time, then averages across tasks and benchmarks. This is a descriptive measure: because the best performer can differ from task to task, the resulting “frontier” does not describe one agent that achieved all those results.

In the authors’ analysis, function-calling benchmarks were close to saturation, programming showed the largest improvement over the period examined, and enterprise workflows had the lowest frontier score among the five groups. These patterns describe the benchmarks and records included in Messier; they should not be read as a complete measurement of progress across all AI systems or real-world work. Benchmark coverage, scaffolds, and data sources differ between groups, and the paper explicitly cautions that this analysis is not an estimate of field-wide algorithmic progress.

The authors also tested how scoring rules affect measured performance. For three benchmarks with multiple verifier results per trial, they rescored the same attempts using partial credit and a rule requiring at least half the checks to pass. The alternative scores were substantially higher than strict all-pass scores in some cases. For example, on HarveyAI-Lab, the paper reports an all-pass rate of 0.17% alongside a mean verifier score of 29.27% and a 50%-threshold success rate of 30.28%.

That difference does not mean the agents suddenly performed better: the underlying verifier results stayed the same. What changed was the rule for counting a trial as successful. In this sample, agent rankings remained strongly correlated between all-pass and partial-credit scoring. The authors say more benchmarks and agents are needed to establish when a scoring change also changes comparative rankings. Their practical recommendation is to report verifier-level results when available, rather than only the scored outcome of each trial.

What the evidence can—and cannot—say

Messier’s records also supported capability estimates that correlated with Epoch’s published overall rankings (Spearman correlation 0.84 for the matched set described in the paper). This suggests that a shared analysis of existing data can recover a broadly similar ordering in this comparison; it does not show that the two systems measure exactly the same thing. The authors also found that task descriptions carried some signal about task difficulty, especially for comparisons within a benchmark, while noting that predicting difficulty remains challenging.

There are important limits. Messier inherits the design and quality of the benchmarks it collects and does not independently revalidate every task. The corpus is concentrated in technical and professional knowledge work, is mostly English-language, and reflects largely Western software conventions. Some upstream data have restricted access or mixed licensing conditions, so the collection is not one uniformly licensed dataset. The authors also caution that some rescoring comparisons involve relatively few agents.

Finally, this is a preprint, so its claims should be treated as research findings that may change with scrutiny and replication. Messier is best understood as infrastructure for asking more detailed questions—not as a final verdict on what AI agents can do. Its clearest lesson is methodological: when a trial is judged by multiple checks, the scoring rule is part of the result, not a neutral footnote.