When AI error labels reveal more about the judge than the mistake

Madhu Sudhan Rao Beesetty’s study began as an attempt to measure a particular reasoning failure across models and languages. It became an investigation of why that measurement did not hold up. The paper is available as an arXiv preprint; its metadata reports acceptance at IEEE CogMI 2026.

The central category, Compositional Collapse, describes a model explicitly noticing a required part of a problem, then dropping it or misapplying it later. Imagine recognizing that a rate changes halfway through a journey, but subsequently calculating as though it stayed constant. That differs from never understanding the condition—or making a small arithmetic mistake.

The distinction is appealing. Applying it consistently proved harder.

Which mistake gets counted first?

The study analysed 2,316 graded records from twelve models across eleven languages, using the public mGSM-Symbolic math benchmark. The battery contained 22 problem instances, but only eight underlying problem structures: many records were variations or translations, not independent problems.

Models solved each question in isolation, without tools. A fixed inspector model, identified in the paper as Claude Sonnet 5, then assessed the answer and collected textual evidence for several failure categories.

Those evidence fields were independent. The final classification was not: it assigned one category by checking them in order and stopping at the first match.

That choice mattered because 50.1% of failures carried evidence for multiple categories, overwhelmingly Collapse together with another category called Reasoning Drift. Checking Collapse first made it the most frequent failure category in every model. Checking Drift first instead changed the leading category in eleven of twelve models.

Across all six effective category orderings, Collapse’s pooled share of failures ranged from 26.4% to 76.4%. Nothing about the model outputs changed. The classification procedure did.

This does not make overlapping categories useless. It means a single-label chart can conceal a consequential tie-break rule.

Agreement on answers, disagreement on explanations

Native speakers covering all eleven languages reviewed 179 transcripts. Their agreement with the inspector on answer correctness was high: 97.8% across 178 matched records, with Cohen’s kappa of 0.932. Kappa measures agreement after accounting for agreement expected by chance.

For Collapse, agreement fell to 50.7%, or kappa 0.093, on the 142 items humans judged incorrect.

Perhaps the category itself was too slippery. A second independent native speaker per language helped test that possibility. Among 121 items with determinate labels from both human rounds, human–human agreement reached kappa 0.659. Each human round agreed much less with the original inspector. Another 21 second-round responses were marked “Unclear” and analysed separately.

The next experiment changed the inspector’s instructions, not its model. Given the human instructions verbatim, it agreed with the second human round at kappa 0.636 on 106 mutually determinate items. The original prompt achieved 0.147 on those same items.

That points strongly toward the instrument’s design. But it does not isolate one magic wording change: the revised instructions bundled a narrower definition, an example, different framing and response options. The revised inspector also abstained on some cases.

A missing phrase changes the diagnosis

Investigating systematic disagreements uncovered another problem: some translated questions did not match their reference answers.

In two puppy-counting items, the Bengali and Chinese translations omitted the attribute saying that one group of puppies had spots. The intended answer, 89%, required those puppies to count as spotted. The corrupted wording supported 11%, which Bengali model responses supplied.

The inspector nevertheless frequently labelled these responses Collapse. Both human rounds generally did not. A benchmark defect was being counted as a reasoning failure.

In the first human comparison, the inspector assigned Collapse 1.81 times as often as humans overall. Excluding the entire two-problem round containing these defects—not just the damaged translations—reduced that ratio to 1.17. Substantial item-level disagreement remained.

A useful warning, not a universal verdict

The evidence concerns one taxonomy, a narrow math battery and a confounded model panel. Human sampling was uneven, some transcripts were shortened, and no English generation control was included. Human agreement establishes reproducibility under shared instructions, not an independent ground truth. The study also does not establish that aggregate model rankings would change under different raters.

Its practical recommendations are straightforward: report overlapping labels, vary classification order, validate against independent human review, and investigate systematic disagreements at the item level.

A plausible explanation of an AI mistake is not yet a reliable measurement. Sometimes the most revealing error is in the ruler.