Matching Score Distributions Could Help AI Judges Align With Human Ratings

An AI judge can consistently score answers too generously or too harshly, even when its ordering of those answers is useful. Researchers at Upwork investigate whether correcting the distribution of its scores—rather than fitting individual score pairs alone—offers a better way to bring those ratings closer to human judgments.

Correcting the measuring scale

Large language models increasingly evaluate other models’ answers, assigning ratings for qualities such as helpfulness, factual consistency or fluency. Human evaluation remains the reference in this study, but collecting those judgments takes time and resources. Automated judges offer a faster alternative, provided their scoring habits can be brought into closer agreement with people’s.

In an October 2026 preprint, Ilija Subasic, Andrew Rabinovich and Zhao Chen of Upwork examine a correction called streaming quantile matching, or SQM. The judge’s model weights stay unchanged. Instead, a separate mapping learns from a labeled anchor set: examples with both an AI score and a human rating.

The central idea is that ratings belong to an ordered scale and form a distribution. A judge might assign high scores much more often than human raters do. Correcting that pattern requires attention not only to individual errors, but also to how ratings are spread across the available categories.

Matching relative positions

SQM connects a score’s position in the judge’s accumulated score distribution to the corresponding position in the human distribution. Conceptually, it asks: where does this rating sit among the judge’s ratings, and what human rating occupies that position among human scores?

The mapping updates as new labeled examples arrive. On the five-category scales used in the experiments, the empirical distributions can be tracked with running counts rather than storing every observation. The approach adapts classical quantile matching; the authors’ proposed contribution is its application and evaluation for LLM-judge calibration, not the invention of quantile matching itself.

A second version accepts a predictive distribution over possible ratings instead of one score. It pools that probability mass into the judge’s running distribution and maps each supported rating separately. This variant tests whether retaining more information about the judge’s uncertainty improves calibration.

What the comparison found

The main benchmark covers BiGGen Bench, FLASK, HelpSteer2, SummEval and UPHELD, with GPT-5, Kimi K3, GPT-OSS-20b and Qwen3-30B as judge backbones. The paper reports 62,621 raw judgments across the five datasets. Experiments reserve 30% of the data for evaluation, never using that portion to fit the calibration functions, and repeat each setup three times.

The results tables compare 17 calibration methods. Single-score SQM ranks first overall under the authors’ comparison procedure. Its pooled median mean absolute error is 0.53 rating points, a reported 31.37% improvement over raw scores, with an exact-match rate of 0.55.

It does not win every measure. Few-shot exemplar calibration has a slightly lower pooled median MAE, at 0.52, while Dawid–Skene aggregation and isotonic regression reach a median exact-match rate of 0.57.

SQM’s clearest advantage is matching the overall human score distribution. Its median Wasserstein distance is 0.26, while the predictive-distribution variant reaches 0.24, the lowest value reported. This metric measures how much score mass must move, and how far along the rating scale, to match the human distribution. Lower is better.

That distinction matters: a method can reduce average error while still producing an unrealistically narrow spread of ratings. Distributional agreement and accurate judgments on individual answers are related but different goals.

Useful corrections, with boundaries

The authors report unchanged pooled median rank correlations before and after calibration. Their interpretation is that calibration mainly corrects scoring scale and distribution, rather than improving which answers the judge considers better.

Task structure also matters. BiGGen Bench, with instance-specific rubrics covering multiple capabilities, remains difficult for every tested method. The authors argue for fitting separate corrections to distinct attributes or rubrics rather than combining heterogeneous tasks into one mapping.

An expanded model-size analysis finds that calibration can substantially narrow some smaller judges’ error gaps relative to larger judges. But the effect is not universal: Qwen3.5-2b’s gaps widen, and the authors note unusually frequent output-processing failures for that model.

The evidence is limited to English data and five-point rating scales. The practical promise is therefore narrower than replacing human evaluation: a lightweight, task-specific correction can make an existing AI judge’s scores more human-aligned, while still depending on suitable human anchors and a coherent evaluation rubric.