AGORA learns which AI judges to trust when evaluation data is scarce

Combining several AI judges does not automatically produce a reliable verdict. A simple average gives every judge equal influence, including models whose ratings are weakly related to the quality being assessed. In the November 3, 2026 arXiv preprint “AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory,” Junhyung Ahn proposes learning those differences directly from the judges’ scores. The supplied source identifies a version-one preprint and does not establish peer-reviewed publication.

AGORA—Aggregation Of Reliability-weighted Assessments—targets absolute ordinal ratings, such as scores from one to five. Its central claim is that a compact statistical model can estimate judge reliability more effectively than a separate error table for each judge when relatively few evaluated examples are available.

Separating reliability from scale use

The method assigns every evaluated item an unobserved quality value on a shared scale. Each judge receives a sensitivity parameter, called discrimination, and ordered thresholds describing how it uses the rating categories. Those thresholds can represent tendencies such as giving generous scores or compressing ratings into a narrow range.

A highly discriminating judge changes its ratings sharply as modeled quality increases. A weakly discriminating judge provides less information. AGORA uses a regularized horseshoe prior—a statistical mechanism that encourages weak signals toward zero while allowing stronger ones to remain—to reduce the influence of uninformative judges.

For a scale with K categories, the model uses K parameters per judge: one discrimination and K−1 thresholds. The paper contrasts this with K² cells in a confusion-table approach such as Dawid–Skene. The proposed benefit is less fragmentation of scarce observations, not additional human supervision.

The final score comes from each item’s posterior mean quality, mapped back onto the rating scale. Although the paper interprets influence through squared discrimination and item-specific information, the implemented read-out is not simply a fixed weighted average of raw ratings.

What the experiments report

Ahn evaluates AGORA on SummEval, with 1,600 summaries; Topical-Chat, with 360 dialogue responses; and WMT22 English-to-German translation, with 19,725 segments. Human ratings serve as evaluation references, not fitting inputs. The primary panel contains five models selected by name before consulting benchmark-specific human references.

On the full datasets, AGORA’s reported Spearman rank correlations with human ratings are 0.556, 0.705 and 0.426, respectively. Corresponding simple-average results are 0.521, 0.672 and 0.416. Dawid–Skene scores 0.555, 0.695 and 0.423: effectively near parity on SummEval, with larger numerical gaps on the other two benchmarks.

When methods are refitted on shared random subsets of 50 items, AGORA scores 0.550 on SummEval, 0.696 on Topical-Chat and 0.409 on WMT22. Dawid–Skene scores 0.529, 0.678 and 0.399. These results support the author’s small-sample argument within the tested setup. The paper reports paired-bootstrap support for the small-sample gains on SummEval and Topical-Chat; it does not make the same confirmation claim for WMT22 in that discussion.

Additional experiments vary panel composition. In a separate draw of 60 five-judge pools at 50 items, AGORA outperforms Dawid–Skene in 98% of SummEval pools and 100% of Topical-Chat pools. Controlled experiments adding uniformly random judges also show greater stability than averaging, according to the paper.

A diagnostic, not a guarantee

The author introduces “agreement dispersion,” measuring how unevenly judges correlate with their peers. Across the 60-pool experiments, dispersion correlates with AGORA’s improvement over averaging: 0.73 on SummEval and 0.61 on Topical-Chat. This offers a score-only signal of when weighting may help, not a guarantee of correctness.

The assumptions matter. AGORA fits one quality dimension at a time, constrains judge discrimination to be positive, and combines ratings through a likelihood that multiplies judge contributions conditional on latent quality. Explicit modeling of judge dependence remains an extension. Its current likelihood also does not cover pairwise preferences.

Finally, stronger rank correlation does not establish that individual scores are correct or suitable for consequential decisions. Evidence is limited to three benchmark families and the reported judge pools. Ahn explicitly positions AGORA as a meta-evaluation tool, not a replacement for human judgment in high-stakes settings.