What should count in an AI benchmark?

What makes an AI test question worth improving on? A difficult question is not necessarily socially useful, and a useful question may be expensive to improve on or produce unstable results across training runs. A study by Andreas Haupt and colleagues brings these distinctions into the economics of benchmark design.

The paper combines a theoretical model with a computational audit of existing language models and evaluation questions. The supplied arXiv version was revised on November 8, 2026; its record reports acceptance at the NeurIPS 2026 Evaluations and Datasets Track. It does not experimentally show that changing a leaderboard improves workers’ lives or redirects a real laboratory’s investments.

A score is also an incentive

The researchers use a principal–agent model: a benchmark designer chooses how much each question counts, while a laboratory responds by allocating costly training effort. Those investments could include pretraining, supervised fine-tuning on example answers, or reinforcement learning using rewards.

In the baseline theory, effort changes expected performance through a linear relationship, costs rise quadratically, and outcomes contain Gaussian noise. The laboratory dislikes risk. The designer values socially useful improvements but also accounts for training costs and the risk borne by the laboratory.

Under those assumptions, the authors derive an optimal weighting rule involving three ingredients: how performance relates to social priorities, how readily investments improve it, and how variable the measured outcomes are. In the simplest one-dimensional case, greater usefulness and easier improvement support a higher weight; greater noise reduces it, all else equal. The full model also allows one training activity to improve several questions together.

This is not a universal recipe with values supplied by mathematics. Someone must still decide which outcomes matter and how to represent risk. The empirical study audits proxies for the three ingredients rather than recovering everything needed to calculate a single definitive set of weights.

Whose priorities enter the calculation?

For social value, the authors deliberately adopt a worker-centered perspective. They use WORKBank, which contains preferences collected from 1,500 workers about tasks they would like AI to automate and how much human involvement they want retained.

The researchers connect benchmark questions to those preferences through 24 broad cognitive work activities from the O*NET occupational taxonomy. A language-model judge ranks the five activities most relevant to each question, separately considering autonomous AI and AI assisting a person. Higher-ranked activities contribute more to the resulting preference score.

Two human annotators checked the mapping procedure on 28 questions. Agreement was imperfect between humans as well as between humans and the model. These scores therefore represent an interpreted connection to worker preferences—not measured workplace benefits.

The audit finds that only 6% of automation-framed loading mass falls on activities where workers welcome AI under both automation and assistance framings. That percentage describes the mapping’s allocation across activities, not the fraction of workers who support AI.

In a separate welfare comparison covering 317 questions, all 52 math questions were outperformed on both preference dimensions by at least one general-knowledge question. None appeared on the welfare Pareto frontier: the set for which no other question scores at least as highly on both dimensions and strictly higher on one. This result concerns the sampled questions under this particular worker-preference mapping. It does not establish that mathematics is socially unimportant.

Cheap gains and unstable answers

To estimate improvement costs, the team evaluated checkpoints from the four-billion-parameter EvoLM family. These vary pretraining, supervised fine-tuning or reinforcement-learning effort around an anchor configuration. The authors fitted question-level relationships between investment and performance, then translated effort into estimated dollars using public pricing assumptions.

Supervised fine-tuning was the cheapest improving route for 50.3% of eligible questions, reinforcement learning for 35.4%, and pretraining for 14.3%. Within the model, equal weighting can consequently favor capabilities that offer cheap score gains. These are recipe- and price-dependent estimates, not universal training costs or observed laboratory decisions.

For variability, the researchers used a different model family: PolyPythias at 410 million parameters, with eight training seeds and ten checkpoints. HellaSwag, PIQA and ARC-Challenge were disproportionately represented among noisier questions. HellaSwag’s elevated variability persisted after adjustment for average correctness. Closely competing answer options offer a possible explanation, rather than an experimentally established cause.

An audit, not a replacement leaderboard

The principal limitations sit at the connections between these measurements. Costs and variability come from different model families, both far smaller than frontier systems. Worker preferences omit consumers, students, safety effects and other interests. Welfare rankings also change with the language-model judge and aggregation choices, while relationships among questions complicate statistical interpretation.

The contribution is a framework for making benchmark trade-offs explicit. Equal weighting is itself a design choice. Assessing that choice requires asking not only what a test measures, but what improving its score rewards.