AI complaint decisions show small group biases—and a separate wording lottery

Colin Priest of UNSW Sydney reports the findings in a preprint posted to arXiv on November 6, 2026. The study examines linguistic profiling: whether a system responds differently to styles associated with age, education or socioeconomic status, even when the underlying complaint is intended to remain unchanged.

Testing different voices

Priest used narratives from the US Consumer Financial Protection Bureau’s public complaint database. Two language models independently rewrote each complaint into 36 combinations of age, geographic setting, education and socioeconomic style. Instructions required them to preserve facts, allegations and requested remedies.

After an exploratory analysis of 2,000 complaints, the main confirmatory tests used 1,000 previously untouched complaints, yielding 72,000 rewritten versions across the two models. Main hypotheses were registered before their test data existed and evaluated once on the held-out sample.

GPT-4o-mini and GPT-5-mini assigned one of three outcomes: no action, non-monetary action or monetary relief. Each judged the versions it had generated. A third system, Jev, returned probabilities for all three outcomes and evaluated both rewrite sets. Its answer options were presented in all six possible orders. Repeated scoring and independent paraphrases helped measure variation unrelated to demographic style.

Small averages, inconsistent individual outcomes

Styles representing older, college-educated and higher-status writers generally received slightly more generous outcomes. Nine of 12 registered directional comparisons were supported. Geography had no detectable effect.

However, every group difference in monetary-remedy rates on the held-out sample was within half a percentage point. The largest was GPT-4o-mini’s education gap: 0.47 percentage points, with a 95% confidence interval from 0.08 to 0.83 points.

Individual complaints told a different story. Two style versions of the same complaint received different monetary decisions 3.8% of the time with GPT-4o-mini and 6.3% with GPT-5-mini. Jev’s disagreement rates were 3.3% and 2.6% on the respective rewrite sets, using thresholds chosen to match each GPT model’s overall monetary-decision rate.

These measures are not interchangeable. A group gap records a net advantage: changes in opposite directions can cancel. Individual disagreement counts both directions. Small average disparities can therefore coexist with unstable decisions, without establishing which problem causes greater harm.

A post-hoc analysis attributed about 1.6% of variation within complaints in Jev’s monetary probabilities to persona effects shared across both rewrite models. Much more variation involved the rewrite model, particular paraphrases, answer order and scoring noise. Some components could not cleanly separate paraphrasing from persona effects specific to one rewrite model.

Facts mattered more—but style still mattered

An analysis of writing attributes associated GPT-4o-mini’s education and socioeconomic gaps mainly with delivery, including vocabulary, grammar and politeness, rather than substantive content. This was a statistical association, not proof that particular features caused the decisions.

Positive controls tested whether the systems responded to relevant evidence. For 200 complaints, edits strengthened the case by adding a documented bank error and financial loss, or weakened it by attributing the problem to the customer’s mistake or agreed terms. The combined fact span—the remedy-tier outcome for strengthened complaints minus that for weakened complaints—was 49–387 times the largest persona gap for the corresponding system. These ratios compare the two edits with each other, not either edit’s individual shift from the original. The edits were fact-led, though they also changed some perceived writing attributes.

A partial remedy

For 500 complaints, Priest tested rewriting every version into a plain, neutral, polite and chronological style before assessment. GPT-4o-mini’s education and socioeconomic gaps in remedy-tier scores fell by 94% and 87% in point estimates. These percentages describe score gaps, not monetary-rate gaps, and did not establish complete elimination.

The intervention retained sensitivity to the factual controls but did not establish improved individual stability. For Jev, it reversed the age and education gaps. Averaging scores across three standard styles reduced score-level variation. Separately, a post-hoc analysis used majority voting across those styles to make monetary decisions. Disagreement between two versions fell from 4.1% to 3.6% for GPT-4o-mini and from 2.3% to 1.9% for Jev. These were reductions of 0.5 and 0.4 percentage points in point estimates; their 95% confidence intervals were −0.2 to 1.2 points and 0.0 to 0.8 points, respectively. GPT-4o-mini’s interval therefore included no reduction.

Important limits

The rewrites did not always preserve facts. Two outside LLM auditors flagged material changes in 3.3% and 11.7% of persona rewrites. Instability persisted among versions that passed the screen, but auditor agreement was low and there was no human validation.

There was also no ground truth establishing which remedy was correct. The synthetic styles do not represent samples of real demographic groups. Results concern English-language US financial complaints under one prompt containing historical company-response proportions—not verified measures of deserved relief. Several analyses, including the variance breakdown and materiality benchmarks, were post hoc.

The preprint’s central recommendation is consequently about auditing, not deployment approval: measure group disparities and individual instability separately, evaluate actual decisions rather than just smoother scores, and subject proposed fixes to the same scrutiny.