Estimating AI Reliability From Fewer Repeated Trials

Three questions about repeated success

The preprint, Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation, examines repeated evaluations of a fixed AI system. Its central idea is to estimate how success probabilities vary across questions, rather than treating each question’s observed success rate as its true probability.

The authors distinguish three metrics that answer different questions:

• Pass@k: the probability of getting at least one correct answer in k attempts.
• Pass^k: the probability that every one of k attempts succeeds.
• AUC@k: the average of Pass@r over every integer attempt budget r from one through k.

At one attempt, all three equal average accuracy. Beyond that, they separate finding a successful answer from succeeding consistently, while AUC@k summarizes success across multiple attempt budgets.

Simply inserting each question’s observed success fraction into these calculations introduces bias. For more than one attempt, this naive approach overestimates Pass^k and underestimates Pass@k and AUC@k. Unbiased combinatorial estimators are available, but require at least as many observed trials per question as the target horizon k.

One fitted distribution, several metrics

The proposed method models each question as having an underlying success probability. Repeated outcomes follow a binomial model, assuming attempts are independent once that probability is fixed. Across questions, the probabilities follow a weighted mixture of Beta distributions, a family defined between zero and one.

The researchers fit the distribution by marginal maximum likelihood and use the Bayesian information criterion, or BIC, to choose between one and four components. Once fitted, the distribution provides formulas for all three metrics, including horizons longer than the observed trial budget. These longer-horizon estimates are model-based extrapolations, not additional observations.

Despite the mixture framework, BIC selected just one Beta component in 95% of repeated-task fits. The reported benefits therefore did not generally require a complicated distribution.

Testing extrapolation against fuller records

The authors analyzed released outcomes for 104 model–benchmark groups, each containing 80 recorded trials per task. The collections covered competition mathematics and 3,600 graduate-level SuperGPQA questions. They compared estimates from smaller trial subsets with combinatorial estimates using all 80 trials, averaging results across three subsampling seeds.

With five observed trials and a target horizon of 60, mean absolute error averaged over Pass^60 and AUC@60 fell from 12.90 to 4.84 percentage points on Trace, from 14.77 to 6.55 on Math, and from 13.65 to 1.42 on GPQA.

The authors report lower average errors than naive estimation in all six collection–metric comparisons for Pass^k and AUC@k when the horizon exceeded the observed budget. However, when enough trials were available, the combinatorial estimator was more accurate in all six comparisons. Their recommendation is consequently conditional: use direct combinatorial estimates when observations reach the target horizon, and the fitted model for extrapolation.

Uncertainty without more model calls

The preprint also derives approximate 95% confidence intervals using the delta method, which translates uncertainty in fitted parameters into uncertainty in the metrics. This requires no additional AI responses.

For 20 models on AIME 2025, using five trials per task and a horizon of 60, all 60 intervals across the three metrics contained the 80-trial reference. Average widths were nevertheless substantial: 25 to 30 percentage points, depending on the metric. This demonstrates agreement in that experiment, not universal coverage. The intervals assume the selected component count is fixed and omit uncertainty from choosing it.

Measuring prompt sensitivity

The same framework can treat prompts, rather than questions, as instances. On MMLU, BBH and LMentry outcome matrices, the authors estimated the distribution of prompt scores from partially observed answers.

Their method achieved lower mean distribution error than pIRT, an outcome-only item-response baseline, at every tested benchmark–budget combination. The mean relative reduction was 46%. A separate baseline using external features, X-pIRT, performed better on BBH and at larger MMLU budgets.

What remains uncertain

The model assumes conditional independence and does not account for shared question difficulty. Its 80-trial reference is itself a finite-sample estimate, and repeated-task experiments cover reasoning benchmarks only. Sensitivity experiments vary prompts, not other configurations. The authors also question whether BIC is the best selection criterion.

Finally, the prompt experiments compare fitted latent probabilities with finite-question scores, whose distributions need not coincide. The evidence supports a useful low-budget estimation strategy within these settings, not a replacement for adequately sampled direct evaluation.