Same Words, Different Scores: How Authorship Labels Sway AI Judges

If an AI system is judging writing quality, should a filename change its verdict? Sergey V. Popov’s experiments suggest that authorship cues can shift scores even when every word stays the same—with an important qualification: the main experiments supplied filenames within the prompt, rather than through uploaded files.

A label changes, the writing does not

In the preprint Same Text, Different Filename: Are LLM Judges Biased?, Cardiff University economist Sergey V. Popov investigates whether an apparently peripheral detail can influence automated writing assessments. His central finding is that identical text can receive different scores depending on the authorship suggested by its filename.

The result is not a simple story of AI systems always favoring themselves or always penalizing competitors. Popov reports that the direction and size of these effects depend on both the evaluating model and the particular document. An average close to zero can therefore conceal opposing effects on different texts.

How the experiments worked

The first study used four English-language documents Popov had written without LLM assistance: a Welsh chess overview, a teaching-certificate assignment, an economics exam answer and a public-audience book segment. They ranged from 340 to 2,915 words and, according to the paper, had not previously been available online.

Each text appeared under six filenames: a neutral name, names suggesting Sergey or Chen, and names suggesting ChatGPT, Claude or Gemini. The content stayed unchanged. Three models—GPT-6.1 Sol, Claude Sonnet 5.5 and Gemini 4 Argon—were asked to return only a writing-quality score from zero to ten, using one decimal place.

Each model assessed every document-label combination 152 times, producing 10,944 valid responses. Calls used separate conversations, with document and label order randomized.

A crucial qualification concerns delivery. Popov reports that OpenRouter removed uploaded filenames, so he instead included the filename in the text sent to each model. The main experiment therefore tests filename-like cues within a prompt, not simply renaming an uploaded attachment.

No universal preference

In the human-written sample, the OpenAI and Google models scored LLM-labelled texts lower, on average, than the neutral-or-Sergey comparison specified in the study. Claude showed the opposite pattern. These average differences were roughly 0.1–0.2 points on the ten-point scale.

Nor did all models reward their own brand. Gemini showed a small own-family label premium, while GPT and Claude showed small penalties relative to rival-model labels.

The comparison between human names also resisted a sweeping conclusion. Chen scored about 0.05 points below Sergey for Claude and Gemini, but essentially the same for GPT. Against a combined Sergey-and-neutral comparison, Chen was not consistently penalized. These findings do not establish a general penalty for Asian-coded names.

The larger story emerged within individual documents. Claude’s own-family label effect ranged from a 0.56-point penalty for the book segment to a 0.75-point bonus for the economics answer. Gemini’s Chen-versus-Sergey difference switched from a 0.35-point penalty on the book segment to a 0.24-point bonus on the exam answer.

Testing a broader collection

A second, preregistered study expanded the material to 90 AI-generated texts: three models each wrote responses to 30 briefs spanning academic writing, general-audience essays and reports. Popov added a Catherine label and a version label, yielding eight labels overall. Twelve assessments per combination produced 25,920 valid responses.

Here, all three judges scored LLM-labelled writing below human-name-labelled writing by approximately 0.11–0.13 points on average. Separately, under neutral labels, GPT and Gemini favored their own family’s output relative to the study’s statistical benchmark; Claude showed the reverse.

Popov also separated persistent label effects from document-dependent effects. After correction for finite-repetition noise, persistent effects accounted for about 64% of label-related variance for GPT, 18% for Claude and 63% for Gemini in the generated-text study. Context-dependent effects were therefore especially prominent for Claude, rather than equally dominant across all judges.

What the evidence supports

The study assesses writing quality, not scientific correctness, editorial decisions or the performance of a particular reviewing service. Its first experiment contains only four human-written documents and a small, deliberately selected set of labels.

Delivery matters too. A limited exploratory comparison involving one model and one document produced different label effects across prompt-based and uploaded-file conditions, with differences in harness and context also present.

Popov suggests removing identifying metadata and averaging assessments across models and repeated calls as possible responses. The practical question, he argues, is how costly evaluation mistakes are. His evidence shows why checking average bias alone may miss the label sensitivity affecting a particular submission.