AI reasoning needs more than a single measure of uncertainty

What declining uncertainty can hide

Language models trained with reinforcement learning can receive rewards for producing answers that an automated checker accepts. In mathematics, that checker can compare a model’s final answer with a reference answer, without evaluating every step of its reasoning.

Researchers often monitor policy entropy: a measure of how spread out the model’s probabilities are across possible outputs. Lower entropy means a more concentrated distribution. But concentration could mean fewer mistakes, less varied wording, or fewer distinct correct approaches.

In a preprint posted on November 6, 2026, Song Yu, Li Li and Zhenming Wu of Southwest University propose separating these possibilities. Their Reward-Conditioned Entropy Allocation framework, or RCEA, is a diagnostic approach rather than a new training algorithm.

Its central message is that uncertainty should not simply be maximized or minimized. Researchers need to ask which uncertainty changes—and what behavior disappears with it.

Separating answers, steps and strategies

The framework first distinguishes uncertainty over complete generated solutions from the average next-token entropy commonly logged during training. These are different mathematical quantities: averaging uncertainty at individual generation steps is not the same as measuring uncertainty across whole responses.

It then separates generated solutions according to whether their final answers pass the verifier. Crucially, an accepted answer does not establish that every intermediate step is valid.

For accepted solutions whose reasoning also passes a process-validity assessment, the framework groups responses by their core mathematical strategy. Different wording or presentation can belong to the same strategy. This lets researchers distinguish losing alternative valid methods from merely standardizing how one method is expressed.

The study summarizes strategy diversity using an effective mode count derived from the distribution’s entropy. A decline means solutions have become concentrated among fewer effective strategies, not necessarily that other strategies have become impossible to generate.

Comparing six training objectives

The team trained Qwen2.5 models at two sizes, three billion and seven billion parameters, using six reinforcement-learning objectives. Each run used one pass through the same 8,000 training examples, with eight generated completions per prompt. Shared training choices were held fixed within each model size.

Evaluation included five mathematical benchmarks, alongside MATH500 pass@64, which estimates coverage when a model has 64 sampled attempts rather than one. A fixed set of 200 probe questions supported uncertainty and strategy-retention analyses. The semantic grouping procedure underwent an independent human audit.

Overall entropy did not provide a stable ranking of reasoning performance. For example, GTPO pushed training entropy close to zero early, yet its final MATH500 greedy accuracy remained 61.0% for the smaller model and 71.0% for the larger one. Preserving more uncertainty did not consistently produce better scores either.

Failed responses generally showed greater average token uncertainty than successful ones. Because separately aggregated groups could contain different questions, the researchers also compared both outcomes within the same eligible questions. All 12 displayed endpoint estimates showed higher entropy for failures; 95% confidence intervals excluded equality in 10 cases. This comparison covered only questions with enough examples of both outcomes.

Strategy breadth and local changes

Every reportable trained endpoint showed some estimated contraction in effective correct-solution diversity. Retention relative to the starting model ranged from 83.7% to 99.9% at three billion parameters and 85.9% to 96.2% at seven billion.

Greater retention was positively associated with pass@64 across the small sets of compared methods. The authors treat this as exploratory, not proof that preserving strategies causes better repeated-attempt coverage. Strategy retention also did not identify the best single-answer performer.

A separate analysis examined how changes in token scores relate to changes in entropy. A local covariance approximation matched the direction of measured entropy change in 42 of 44 adjacent-checkpoint comparisons from nonfailed runs. This was a diagnostic calculation using checkpoint differences, not a demonstration that the method can forecast future training behavior.

A diagnostic, not a causal verdict

The evidence comes from one training seed and one model family at two sizes. Process validity remained model-judged, despite human auditing, and the paired-outcome and strategy analyses covered restricted question sets.

The experiments also did not measure every component of the theoretical framework, including variation within individual strategies. They do not establish that every observed contraction is harmful.

The practical conclusion is narrower: a falling entropy curve is insufficient evidence of either successful learning or damaging collapse. It needs to be interpreted alongside answer correctness, reasoning validity and the breadth of valid strategies still available.