When language-model training rewards cost more than they teach

The cost of a more detailed signal

In reinforcement learning with verifiable rewards (RLVR), a language model generates a solution and receives a reward for getting the final answer right. Group-relative policy optimization (GRPO), a commonly used method, gives all the tokens in a solution the same credit. That makes it hard to distinguish a useful intermediate step from an unhelpful one.

Dense process rewards aim to fill that gap by scoring steps along the way. One example estimates a step’s value by sampling possible continuations from it. But those extra continuations take computing resources. Comparing methods after the same number of training updates—or testing them on a single batch—may therefore make the more expensive method look better than it is for a fixed budget.

Comparing like with like

The authors introduce a matched-compute protocol: interventions receive the same budget, measured in generated tokens and training GPU-hours. They report preregistered studies with controls for side effects, covering 1,668 runs across six models. The evidence available here is the paper’s abstract, which does not provide enough detail to assess the full experimental setup or every control.

What the abstract reports

In the authors’ setting, continuations used to calculate the Monte-Carlo process reward accounted for 91% of all generated tokens. On Qwen2.5-1.5B, that reward performed better than standard GRPO when compared per update, but finished 14.3 percentage points lower at equal token budgets and 7.8 points lower at equal training time. None of four tested dense-reward variants beat GRPO when given the same training time.

A per-token gradient test ranked the rewards in the same order as training did, but overstated each by one to two points—potentially as much as the gains the test was intended to identify. The abstract does not specify the benchmarks or uncertainty intervals behind these figures, so they should be read as reported results, not universal estimates.

The authors also report gains from training on prompts for which predicted success was closest to 50%. Accuracy rose by 1.7 points on Qwen3-0.6B and 0.8 points on Qwen2.5-1.5B. They say these gains repeated with fresh random seeds and unseen problems. A simple rule—candidate prompts were predicted to fail more often than not—identified where this approach appeared in five of six preregistered tests. The gains came early in training; standard GRPO with twice the initial learning rate came within half a point.

A qualified takeaway

The paper’s central point is about evaluation: a training method should be judged against alternatives given the same resources, not just the same number of updates. Its abstract supports that argument for the tested settings, but it cannot establish that dense rewards are unhelpful in every task or that prompt selection will reliably help elsewhere. This is an arXiv preprint, and the supplied evidence does not establish peer review. The full paper is needed to examine the datasets, implementation details, and statistical uncertainty.