When AI labels political texts, better scores may not mean better rule-following

Separating the rules from the labels

Social scientists use codebooks to turn concepts such as protest types or policy positions into categories that can be assigned to texts. The study asks whether language models respond to those written rules when the rules disagree with familiar label associations or labeled examples.

The researchers used three English-language political-text datasets: U.S. protest news from the Crowd Counting Consortium, reports of political violence in Pakistan from BFRS, and party-platform sentences from the Manifesto corpus. Mistral-7B was the primary model. Different experiments also used Llama models, including an 8-billion-parameter model for fine-tuning and a 70-billion-parameter model for retrieval comparisons.

Their central audit moved each category definition to a different output label, leaving its substantive content unchanged. A model following the supplied mapping should therefore return the newly assigned label rather than the original human annotation. This was a diagnostic stress test, not a proposed replacement classification system.

The main audit used 1,000 documents per dataset and ten rearrangements. Predictions were recorded as matching the reassigned definition, matching the original label, choosing another valid label, or producing an invalid response.

Original mappings proved more persistent

With reassigned definitions and no labeled demonstrations, Mistral matched the new mapping on average in 10.75% of protest documents, 12.87% of political-violence reports and 3.24% of Manifesto sentences. Agreement with the original labels was higher: 55.80%, 20.93% and 30.25%, respectively.

These results do not mean every remaining prediction preserved the old mapping. Many matched neither mapping, particularly on BFRS and Manifesto.

The researchers then fine-tuned Mistral-7B and Llama-3.1-8B on each dataset’s original human annotations. Original-label agreement under the audit increased in all six model–dataset combinations, by 4.63–29.44 percentage points. Adherence to reassigned definitions did not improve consistently: it decreased on the protest dataset for both models, increased reliably only for Llama on Manifesto, and showed uncertain changes elsewhere.

Fine-tuning also improved performance under the original codebook. The distinction is important: learning to reproduce existing annotations was not equivalent to becoming more responsive to a revised definition–label assignment.

Nearby examples supplied much of the benefit

The retrieval experiments compared two ways of selecting demonstrations. Global retrieval chose the training texts most similar to the document being classified. Class-wise retrieval selected examples separately within each category, providing broader category coverage.

Global retrieval generally performed better, including at matched example counts on Manifesto and BFRS. For Mistral, adding globally retrieved examples to a codebook prompt raised test weighted F1 from 0.404 to 0.551 on Manifesto and from 0.548 to 0.774 on BFRS. Weighted F1 combines precision and recall while giving more common categories greater weight; these values are not percentages of documents classified correctly.

A simpler comparison classified each document using the most frequent label among the same retrieved neighbors, without an LLM. Relative to codebook-only prompting, this nearest-neighbor method recovered 74–93% of Mistral’s retrieval improvement on Manifesto and BFRS across development and test evaluations. That suggests the neighborhood’s labels already contained much of the useful predictive signal—not that retrieval made the model better at interpreting definitions.

Conflicting examples could redirect predictions

The researchers also changed demonstration labels while keeping retrieved texts and their order fixed. Predictions shifted toward the substituted labels. Supplying the original codebook reduced this influence in several comparisons but did not consistently override it.

Conversely, aligning demonstration labels with reassigned definitions substantially improved performance against the new target mapping. In a two-label swap on BFRS, Mistral’s test weighted F1 rose from 0.370 with conflicting demonstrations to 0.739 with aligned demonstrations. Performance nevertheless usually remained below that achieved under the original mapping.

What the audit cannot establish

The findings cover three English-language political datasets and selected open-weight models. They do not establish how other models or domains behave. Reassigning labels also tests responsiveness to a changed mapping, not learning genuinely new category boundaries or handling ambiguous wording and isolated annotation mistakes.

Agreement with a mapping reveals observable behavior, not the model’s internal reasoning. Likewise, global and class-wise retrieval differ in both relevance and label concentration, so their comparison cannot cleanly separate those influences. Removing examples from the same manifesto reduced performance, and retrieval benefits were uneven across categories, including rare ones.

The authors’ practical conclusion is to validate definitions, label names, demonstrations, retrieval and fine-tuning data together. Conventional performance checks and controlled rule-following audits answer different questions; neither should be treated as a substitute for the other.