When AI gets a label wrong, the rulebook may need an audit

Researchers increasingly ask large language models to assign categories to text: what kind of violent event a report describes, for example, or who receives credit for a political agreement. A codebook defines those categories and explains their boundaries. But disagreement with a human reference label does not automatically mean the model failed.

In a preprint submitted to arXiv on 6 November 2026, Yuehong Cassandra Tai and Qunhua Li propose auditing the codebook alongside the AI annotator. Their central distinction is between applying a clear rule incorrectly, following an unclear rule, and disagreeing with a questionable reference label.

Looking beyond one score

F1 combines precision—how often a predicted category is correct—with recall—how many reference examples of that category the system finds. It measures agreement, not the reason for disagreement. An overall score can also conceal problems in particular categories.

The proposed audit first screens category-level performance, then uses a confusion matrix to locate recurring mistakes between categories. Researchers inspect five affected documents per error pattern, expanding to ten when needed. They compare the text, coding rules, reference label and model prediction; model explanations are supplementary, not decisive evidence.

Only recurring defects that can be repaired by clarifying the existing rules trigger revisions. Clear model mistakes, questionable reference labels and disagreements requiring a new substantive criterion are documented separately. The same explicit wording problem must recur in a majority of reviewed cases before becoming a revision target.

Revision targets, order and wording are determined from development data. Separate held-out documents evaluate the resulting changes without directing subsequent revisions. Model and annotation settings remain fixed.

Giving “Other” a clearer boundary

The first application used the BFRS codebook, which sorts reports of political violence in Pakistan into twelve event types. The researchers used 573 development documents and 1,045 held-out documents, with Llama-3.3-70B-Instruct as the main annotator. A retrieval step selected relevant codebook entries for each report.

The fallback category Other performed worst on the development sample. Review identified an open-ended assignment rule and a conflicting boundary with Assassination. Although assassination required deliberate targeting, a clarification also included bodies found with signs of violence without explicitly repeating that requirement.

The revisions specified when insufficient information or an event outside the substantive categories warranted Other. They also required evidence of deliberate targeting before an unexplained violent death qualified as Assassination.

On held-out data, Other F1 rose from 0.429 to 0.535. The estimated gain’s 95% confidence interval was 0.031 to 0.185. Riot F1 also improved, although its entry was untouched. Overall macro F1, which gives each category equal weight, changed little: 0.731 to 0.734. This was a targeted improvement, not a broad performance leap.

Another weak category, government attacks on civilians, was not revised. Review found similar reports receiving inconsistent reference labels, pointing toward a reference-label issue rather than a demonstrated wording defect.

Future tense is not the same as no credit

The second application classified statements about European Council summit outcomes into four categories: Credit-claiming, Credit-sharing, Blame-shifting and Descriptive. It used 172 development documents and 321 held-out documents, with the full codebook supplied to the model.

The model assigned too many attribution statements to Descriptive. The codebook included desired actions expressed with “must” or “should” under that category, while other guidance allowed future-oriented credit attribution.

The revision clarified that future-oriented language alone does not make a statement descriptive: who receives credit matters. Held-out Credit-sharing F1 increased from 0.589 to 0.653, with a gain interval of 0.017 to 0.115. Descriptive F1 fell from 0.623 to 0.605 as precision increased and recall declined. Improving the diagnosed boundary did not require every category’s F1 to rise.

A repair can depend on the model

Tests with Gemma-4-31B showed both transfer and limits. The BFRS revisions improved its held-out Other performance too. But a separate clarification about jointly claimed political credit, developed from Gemma’s errors, improved Gemma’s Credit-sharing F1 by 0.044 while reducing Llama’s by 0.029, relative to the preceding codebook version.

The evidence comes from two codebooks and two models, not a universal validation. Higher agreement also does not establish that a revision better represents the intended social-science concept. Independent human application of the revised rules would provide a stronger reproducibility test; domain experts still decide what categories should mean.

This account uses the paper’s extracted text. Images were absent, and extraction may damage tables, equations and layout. The source is a preprint, not presented here as peer-reviewed evidence.