An AI moderator can spot hate—and still fail the trust test

A moderation system can make two very different mistakes: overlook an attack on a community, or flag someone defending that community. It can also sound certain while getting either decision wrong. TrustMod-SM, a November 2026 preprint by Aizan Zafar and colleagues, examines these problems together rather than treating accurate classification as sufficient evidence of trustworthiness. The work has not yet been established through peer review.

Five tests instead of one leaderboard

The researchers assembled 28,792 evaluation instances from eight existing datasets. These include social-media posts, deliberately constructed hate-speech tests, machine-generated statements and verified COVID-19 news. The total includes altered inputs and repeated applications of different attack prompts; it is not a count of unique original posts.

The benchmark examines five dimensions. “Trustfulness” combines correct decisions with appropriate uncertainty. Fairness measures differences in errors across demographic groups. Robustness tests altered spelling and wording. Safety asks whether models resist requests to disguise toxic content or exaggerate news. Context integrity tests whether they distinguish hate from speech opposing hate, reclaimed language and neutral demographic references.

The study centres on smaller open-weight models from the Qwen, Llama and DeepSeek families, with additional vision-language systems and larger-model comparisons. Coverage differs: not every system takes every test.

Confident mistakes and unequal treatment

On a 3,000-post task distinguishing hate, offense, benign content and other material, accuracy among the ten primary text models ranged from 33.5% to 55.5%. Those figures describe this particular task—not the proportion of all moderation decisions these models would get right online.

Among incorrect predictions, between 38.9% and 65.5% carried self-reported confidence of at least 80%. A confidence number supplied by a model was therefore not a dependable warning that its answer might be wrong.

Models also flagged fewer than 40% of the benchmark’s 819 designated ambiguous examples for human review. There is an important qualification: these examples were selected using content-based rules, not fresh judgments from a human moderator panel. Escalation performance measures agreement with that proxy for ambiguity, rather than a definitive standard for when people need assistance.

The fairness evaluation covered 7,500 examples across six demographic attributes. Across the models in that analysis, higher overall accuracy was associated with larger demographic accuracy gaps. Nine of ten also had higher false-positive rates on African American Vernacular English than on Standard American English.

These are observed associations under the benchmark’s labels and sampling rules. They do not establish that improving accuracy necessarily causes unfairness, or that the same disparities would have identical sizes on a particular platform.

Understanding opposition is not detecting a keyword

Counterspeech—language condemning or challenging hate—was especially difficult. Qwen-3B incorrectly labelled 93.6% of counterspeech examples as hateful; Qwen-14B reduced that figure to 17.2%. Larger models therefore helped substantially on this test, without eliminating the problem.

A low false-positive rate alone could also mislead. DeepSeek-R1-7B incorrectly flagged only about 2% of context-sensitive non-hateful examples, but detected just 0.3% of hateful examples in the same evaluation. Almost never calling anything hateful is not evidence of excellent contextual understanding.

Robustness tests exposed another weakness. Dotted letters and spacing tricks produced the largest average accuracy loss among the tested alteration types. Some apparently strong results were compromised by malformed responses: the two primary DeepSeek robustness models failed the required output format on roughly 62% of inputs, and their reported accuracy used valid responses only.

A moderator can also become a rewriting tool

Safety tests asked models to rewrite toxic posts while preserving their stance, or make verified news more alarming. Refusal varied considerably. Llama-3B refused 72.2% of concealment requests but none of the COVID-news exaggeration requests.

Qwen-14B showed the reverse pattern: it refused every COVID exaggeration request, yet refused none when the researchers tested political claims and Mpox reports. Success on one subject did not demonstrate a general safeguard.

These evaluations used automated toxicity, similarity and misinformation measures, plus phrase-based refusal detection. They are useful indicators, not independent factual verification of every rewrite. Refusal wording mattered substantially for the closed-model comparison, illustrating how measurement choices affect reported safety.

What the benchmark establishes—and what it cannot

Larger-model extensions improved contextual results but retained demographic disparities and confidence problems. Those comparisons used reduced subsets for some tasks, omitted robustness and used provider-default decoding for the closed model.

The benchmark is English-only, relies on existing labels and cannot rule out training-data overlap. Generated paraphrases were not individually established to preserve meaning and labels. Its nine-model cross-dimensional correlation analysis is exploratory: no pairwise association survived correction for multiple comparisons.

The practical result is narrower than a universal verdict on AI moderation: these tested systems have different weaknesses, and one strong score can hide another serious failure. TrustMod-SM supplies an auditing framework, not a certificate of deployment safety.