Why AI Reasoning Training May Favor Familiar Problems

Sharpening skills or expanding them?

Reinforcement learning with verifiable rewards, or RLVR, trains language models using outcomes that can be checked, such as whether an answer is correct. But does it help models solve genuinely new problems, or mainly make answers they could already produce more likely?

The arXiv preprint On the Primacy Bias in RLVR Training, posted November 2, 2026, proposes an explanation for apparently conflicting findings. Phuc Minh Nguyen and colleagues argue that models often rely heavily on representations learned before reinforcement learning begins. They call this primacy bias: earlier learning can constrain later adaptation.

Their results support a trade-off, not a universal verdict. Training on difficult problems sometimes increases the range of problems solved, while extended training can undermine previously reliable skills.

What counts as broader ability?

The paper distinguishes accuracy on one attempt, called Pass@1, from Pass@k, which measures success at least once within a specified number of attempts. Pass@128 therefore assesses coverage with up to 128 attempts, not ordinary single-answer reliability.

These measures can move differently. A model might become more dependable on familiar problems while losing the occasional successful answer to others. Conversely, it might begin solving previously unsuccessful problems while becoming less reliable on easy ones.

Importantly, “previously unsolved” means no successes were observed under the evaluation procedure. With finite sampling, that does not establish that the original model had absolutely no chance of solving a problem.

Testing the pull of earlier learning

The researchers first used a controlled handwritten-digit classification experiment. They varied how frequently each class appeared during pretraining, then applied reinforcement learning to distributions emphasizing common classes, rare classes or all classes equally.

In these experiments, pretrained models showed coverage shrinkage, whereas randomly initialized models did not show the same pattern. Training on rare, harder classes could initially improve coverage, but longer training produced losses associated with forgetting common classes. Stronger pretraining imbalance also delayed hard-class learning and produced sparser parameter updates.

The authors connect these observations to a lazy training regime. Here, “lazy” describes limited change in internal representations, not an absence of learning: the model adjusts while remaining close to its starting point.

They then examined publicly released language-model checkpoints. First-order linear approximations around the starting models closely matched the studied RLVR models, with reported divergence values between 0.042 and 0.125. Two supervised-fine-tuning comparisons had much larger values, 12.73 and 17.946. Other measurements also indicated smaller representation changes after RLVR.

These findings are evidence for the authors’ interpretation in the examined settings, rather than proof that every RLVR model follows the same dynamics.

Hard problems helped—but not everywhere

For their main reasoning-training experiments, the researchers used Qwen2.5-3B-Instruct and the MATH dataset. Difficulty was model-dependent: problems were grouped using estimated accuracy after an initial fine-tuning run. Problems with zero accuracy under that final policy were excluded from this difficulty-partitioning analysis.

Easy subsets showed faster, larger training improvements. Hard subsets improved much less, even with focused training. Nevertheless, hard-only training sometimes delivered better evaluation coverage than training on easy problems or the full dataset.

After 512 training steps, OlympiadBench Pass@128 rose from 68.77% for the base model to 69.87% with hard-only training. On MATH-500, it increased from 94.31% to 94.51%. But on the evaluation set labeled Minerva, it fell from 53.91% to 52.98%. Hard-only training therefore did not improve every benchmark.

Extending training to 2,048 steps generally reduced overall coverage. Yet coverage on problems with no observed base-model successes increased under hard-only training. The authors interpret this as new successes being masked by forgetting elsewhere.

Additional Llama-3.2-3B-Instruct experiments showed a similar distinction: hard-only training outperformed full-dataset training on the reported coverage measures, but remained below the base model on two of three benchmarks.

What the findings leave open

The authors suggest that forgetting matters less on unfamiliar tasks where the starting model performs poorly. This could help explain why some earlier studies reported expanded reasoning capabilities. It remains an interpretation, not a general demonstration across unfamiliar domains.

The paper acknowledges finite evaluation sets and task-specific rewards. Its checklist reports no repeated training seeds per configuration and no statistical-significance assessment, limiting confidence in small numerical differences. Controlled reasoning experiments concentrate on mathematics; checkpoint analysis includes coding models, but further coding-training experiments were left for future work because of computational cost.

The preprint offers evidence about adaptation and forgetting, not a proven recipe for overcoming them. Its central implication is that evaluation should distinguish greater reliability on familiar problems from wider coverage—and measure what the model loses as well as what it gains.