When an AI’s training score rises but its answers get worse

Learning from attempts

In What Broke & Why: Practical Lessons from Reinforcement Learning Post-Training, Luv Verma examines how training can fail even when the software appears to run normally. The book-length arXiv preprint follows programs involving mathematics, search agents, training stability, mixture-of-experts models and software engineering.

Its focus is reinforcement learning with verifiable rewards: a model attempts a task, and a program scores the result. A mathematical answer can be checked against a known value; a proposed software repair can be tested. Training then makes higher-scoring attempts more likely.

That differs from supervised fine-tuning, which trains a model on demonstrations of desirable responses. In the book’s workflow, demonstrations can prepare a model to produce usable attempts before reinforcement learning begins. The experiments concern open models and relatively small computing setups, rather than establishing a recipe for every model or task.

Why identical failures teach little

A central method is Group Relative Policy Optimization, or GRPO. The model produces several attempts at the same question. Each receives a reward, and training compares that reward with the group’s average.

An above-average attempt supplies a signal to reinforce its choices; a below-average attempt supplies the opposite signal. But if every attempt receives the same score, those comparisons provide no preference between them. The reward-driven part of the update disappears for that group, although other training terms may remain active.

Verma reports an early run that earned exactly zero reward for 156 steps. The task was too difficult for the model to produce successes worth reinforcing. Waiting longer did not address the missing signal.

The practical response is to inspect groups of attempts before committing to a long run. Are they all wrong, all right, or mixed? Easier starting tasks, demonstrations or a curriculum can help when failures are unanimous. Harder examples may be needed when every attempt already succeeds.

Partial credit brings another risk

Giving partial credit can distinguish attempts that would otherwise all count as failures. For search answers, an overlap score can reward a response that contains some of the expected answer. For an image-localization task, overlap between a predicted rectangle and the reference rectangle provides a graded measure.

But a more informative score is not automatically a more trustworthy one. The book describes a search-agent run in which the reward remained around 0.50 while exact-match accuracy declined from 26.8% to 19.9%. That divergence illustrates why developers need to measure the intended outcome separately from the training reward.

One response explored in the book is to make supplementary rewards depend on correctness. Desirable formatting or tool-use behavior should not become an independent route to a high score while the answer remains wrong. Such gating addresses that incentive problem; it does not make an initially unsolvable task reachable by itself.

Sometimes the measurement is broken

Not every flat reward curve reflects a learning failure. In a small arithmetic-puzzle experiment, a 256-token output limit cut off every attempt. Increasing the allowance to 1,024 tokens brought truncation to roughly 3% in the reported working version. The problem was unfinished answers, not necessarily the learning algorithm.

Evaluation has similar traps. A parser that grabs the last number in a response may extract an intermediate calculation rather than the final answer. A short output budget may stop a reasoning model before it states its conclusion.

The book’s GSM8K reference run offers a concrete positive result: reported accuracy on grade-school mathematics rose from 81.7% to about 94–95%, then plateaued. That is evidence of improvement on that benchmark under the stated evaluation, not proof of broadly expanded reasoning ability.

A debugging record, not a universal law

The preprint combines tutorials, reported experiments, operational advice and hypotheses. These carry different evidential weight. Its appendices explicitly correct earlier claims, including a mislabeled quantization method and results affected by small-sample evaluation.

This account draws on the supplied extracted excerpts, whose middle section is omitted; underlying checkpoints and raw evaluation files were not examined here. The reported outcomes therefore remain the author’s findings, not independently reproduced results.

The strongest practical lesson is narrower than any claim that one training method always works: inspect what the model generated, what the reward actually measured and whether the same evaluation was applied to the starting and trained models. A rising score is useful evidence only when those checks support its meaning.