When AI answers stay the same but people judge them more harshly

The question behind “AI got worse”

When people say an AI assistant has become less useful, they may be describing a real change: companies sometimes update systems without changing their names. But judgments can also change while the product stays put. After encountering better answers elsewhere, a person might find yesterday’s answer less acceptable than before.

Zihan Chen, Lei Nico Zheng and Di Zhu investigate this possibility using Chatbot Arena, a public platform where visitors compare two anonymous AI answers to a prompt. Voters choose the better answer, call it a tie, or say both are bad. Model identities are revealed only after voting. The researchers ask whether verdicts on the same unchanged models became harsher over time—and whether the shift came from voters, prompts, model serving or the site itself.

Comparing like with like

The study analyzes 1,260,967 blind votes cast between May 2023 and August 2024 on pairs of “frozen” models: dated versions published by vendors and models with fixed open weights. It excludes model names that might point to changing versions. The researchers compare votes within the same model pairs, so differences in the models themselves are less likely to explain changes in verdicts.

They also test alternative explanations. The vote log includes information such as prompt language, length and difficulty tags, as well as answer length and whether an answer was flagged as a refusal. The authors compare results across types of models, periods between interface changes, and voters with hashed identifiers. They examine both “both are bad” votes—an absolute judgment of acceptability—and which model wins when a voter picks one answer over the other.

A rising rejection rate, a steadier ranking

Within the same frozen pairs, the share of votes saying “both are bad” rose by an estimated 1.71 percentage points per month over the study period. The average share for those votes was 17.1%. In 14 pairs that were frequently judged in both 2023 and 2024, the share rose in every pair, from 15.5% to 23.3% between the periods compared.

The increase was not matched by a continuing shift in which model voters preferred. In 2024, among votes that chose one answer, the stronger model’s share was essentially flat. The rising “both are bad” share came mainly at the expense of decisive votes; the tie share showed no clear trend. Put simply, voters increasingly rejected both responses, but did not increasingly favor one model over its rival.

The pattern remained when the researchers controlled for recorded prompt characteristics and looked separately at several model groups and major prompt languages. It also appeared within the same voter over time and when comparing the same voter on the same pair. Later-arriving voters were more likely to reject both answers on their first recorded vote than earlier-arriving voters were. These checks make a simple explanation based only on new voters entering the platform less convincing.

Meanwhile, the models appearing on Arena improved by the study’s ranking measure. Yet across all battles, the share of “both are bad” votes ended close to where it began: 19.4% in May 2023 and 20.6% in August 2024. That contrast is consistent with a moving standard: as the broader field advances, unchanged systems may seem less satisfactory.

What the evidence does—and does not—show

The researchers’ interpretation fits the idea that people judge quality against a reference point shaped by experience. They also find that month-to-month shifts are shared across models: using time effects estimated from other models’ comparisons largely removes the trend for a given model. That suggests evaluation platforms could keep fixed reference models in the mix to help track changes in standards, rather than treating every shift in absolute ratings as evidence that a system itself regressed.

But this is observational evidence, not a controlled experiment showing that exposure to better AI made people pickier. The vote log does not record each person’s experience with other systems, and the field’s progress closely coincided with calendar time. The authors therefore cannot separate the influence of better models from other changes occurring over the same period, or tell whether voters raised their acceptance threshold or simply scrutinized answers more closely.

The voters were self-selected Arena users, not a representative sample of the public. Hashed identifiers may track a device or network rather than an individual. The log also lacks the prompt and answer text, so the analysis cannot directly verify that people saw identical prompts and responses across time. The study ends in August 2024; it does not establish whether the trend continued afterward.

The paper is an arXiv preprint, so its claims should be read as research findings that may still be revised or receive further scrutiny. Its central lesson is narrower than the headline: human evaluations can change even when the evaluated versions are held fixed. For AI leaderboards and feedback used in training, that makes the date and context of a judgment important—not just the score itself.