When AI learns from its own answers, agreement is not enough

A language model can generate several responses to the same question and use the most frequent final answer as a provisional training label. This offers a way to adapt to unfamiliar questions without supplying correct answers. But repetition is not verification: a model can become increasingly consistent while reinforcing a mistake.

In a November 6, 2026 preprint, researchers investigate this problem in test-time reinforcement learning, where a model updates its parameters using unlabeled questions and self-generated supervision. Their proposed method, Prompt-Conditional Distributionally Robust DPO, or PC-Dr.DPO, aims to limit the damage caused by sampling-sensitive labels. It does not establish whether a confidently repeated answer is true.

Two different ways a majority can fail

The paper distinguishes between the model’s underlying answer distribution and the limited set of responses actually generated for a question.

The first source of error is population-majority bias: the answer the model is most likely to produce is wrong. Even an enormous sample can then converge on an incorrect answer. The second is finite-rollout variation: a limited sample happens to select an answer other than the model’s underlying most probable answer. Here, a rollout means one generated response.

The theoretical analysis uses a simplified setting with two possible answers, deterministic correct preferences and a fixed model generating independent samples. Under those assumptions, increasing the number of responses reduces disagreement between the sampled majority and the underlying majority when the answer probabilities are separated from an even split. It cannot remove an underlying preference for the wrong answer.

The authors also connect inaccurate preference labels to a bound on prediction risk. The key distinction is that systematic labeling errors can persist even as the training sample grows. These results explain a mechanism under stated assumptions; they are not a general guarantee for unrestricted language-model reasoning.

Making uncertain self-training more cautious

The method builds on Direct Preference Optimization, or DPO. Rather than requiring a separately supplied reward for every response, DPO trains on pairs containing a preferred and a less-preferred response. Here, those preferences come from the model’s own generated answers.

PC-Dr.DPO adjusts the weighting of preference pairs separately for each question. It measures the fraction of generated responses sharing the most frequent final answer and uses that concentration as a proxy for sampling stability—not correctness.

For weaker observed majorities, its weighting concentrates more strongly on pairs whose constructed preferences the current policy already favors. The intended effect is to make learning from unstable self-supervision more conservative. The paper proves that this temperature adjustment reduces an upper bound on the question-level gradient magnitude when the pairwise scores are held fixed, rather than guaranteeing that every actual update becomes smaller.

The complete method also changes answer selection after the first adaptation step. It aggregates trajectory scores based on likelihood ratios between successive model versions, rather than relying only on answer counts.

A further safeguard stores generated completions when every sampled response gives the same final answer. If the majority answer in a later rollout sample conflicts with that cached answer, training uses the stored completions instead of the new preference pairs. This is designed to reduce self-supervised forgetting, but the stored agreement is still not external evidence of correctness.

What the experiments show

The researchers tested five instruction-tuned models from the Qwen2.5 and Llama-3.2 families. Evaluation covered AIME24, AIME25, AMC23, MATH500, GPQA-Diamond and the mathematics subset of MMLU-Pro. The latter contained 1,351 examples; the study did not evaluate the entire MMLU-Pro benchmark.

The authors report that PC-Dr.DPO matched or exceeded the strongest comparison baseline in 24 of 30 model–benchmark settings. Performance was measured at the final checkpoint using Avg@32: average correctness across 32 sampled responses, not the accuracy of a majority-vote answer.

For Qwen2.5-3B-Instruct, the method reached 45.8% on AMC and 73.3% on MATH500, compared with 34.3% and 63.5% for the unadapted model. In a separate Llama-3.2-3B-Instruct experiment on MATH500, removing the cache mechanism lowered accuracy from 58.4% to 56.3%.

Adaptation used the evaluation questions themselves without their correct answers. These results therefore concern adaptation to an available target question set, not demonstrated transfer to an untouched set of future questions.

Agreement remains an imperfect guide

The experiments also expose the central limit. On AIME25, the authors document a small Qwen model becoming nearly unanimous while its pseudo-labels remained predominantly wrong and accuracy collapsed. PC-Dr.DPO also had collapsed runs in the main results.

The preprint offers evidence that handling sampling instability more carefully can improve self-training. It does not show that internal consensus can replace an external check on truth.