MinPRO tests a prefix-aware approach to steadier language-model training

Reinforcement learning can improve a language model’s mathematical answers using rewards based on final-answer correctness, without requiring annotated reasoning steps. A practical complication is that generating responses and updating the model need not happen together. By the time a response is used for training, the model that produced it may already be outdated.

In the arXiv preprint “Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio,” Shiye Lei, Zhihao Cheng and Dacheng Tao investigate this stale-data problem. Their proposed method, Minimum Prefix Ratio, or MinPRO, reports more stable training and stronger average benchmark results in the tested settings. The findings concern mathematical post-training, not a demonstrated improvement across language-model tasks generally.

Why earlier tokens matter

In off-policy training, responses come from an older sampling model while updates change a newer target model. Importance sampling adjusts how strongly sampled data contributes to learning by comparing its probability under the two models.

Many objectives use a token-level ratio: how likely the newer model is to produce a particular token, divided by how likely the older model was to produce it, given the same preceding text. But that preceding text—the prefix—may itself have become unlikely under the newer model.

The paper derives an exact correction for a policy-gradient expression defined over prefixes. It requires multiplying probability ratios from the beginning of the response through the current token. This theoretical result uses the target policy’s advantage, a measure of an action’s relative expected value. It does not establish that a practical algorithm using estimated advantages and clipped weights is exact.

Multiplying many ratios also creates difficulties. The product can become extremely large or small, producing highly variable weights, especially later in long responses.

A deliberately approximate replacement

MinPRO replaces the product over preceding tokens with the smallest individual ratio encountered in that prefix. It multiplies this minimum by the current token’s ratio, clips the resulting weight and uses it to scale the learning update. The implementation gives the first response token a prefix factor of one.

The intuition is that a low earlier ratio signals a mismatch between the sampling and target models. MinPRO retains that signal when weighting subsequent tokens, rather than considering each token’s ratio alone.

This is a biased surrogate, not an exact distribution correction. Avoiding cumulative multiplication removes one source of extreme weights, but does not make the method independent of token position: once a low ratio appears, it can continue to reduce later updates.

What the experiments tested

The researchers trained on DAPO-Math-17K, containing 17,000 mathematical questions with final integer answers. They tested two dense models, Qwen3-8B-Base and Qwen3-14B-Base, and Qwen3-30B-A3B-Base, a mixture-of-experts model with 30 billion total parameters and 3 billion activated during inference.

A rollout buffer imposed controlled delays before generated responses were used. The main setting delayed data by two global training steps, corresponding to 32–47 parameter updates between sampling and use. Responses could reach 20,480 tokens.

Comparisons included GRPO, GSPO, CISPO and M2PO. Evaluation covered seven mathematical benchmarks. The researchers saved checkpoints periodically and selected the one with the highest average benchmark score, so the headline results describe selected peak performance rather than necessarily the final training state.

Gains, with exceptions

Under the main delay setting, MinPRO’s average pass@1 score across the seven benchmarks was 59.4% for the 8B model, compared with 58.7% for M2PO. For the 14B model, the corresponding scores were 62.6% and 62.1%. Pass@1 measures success on a single sampled attempt; multiple generations were used to estimate it.

The reported reward curves showed steadier training for MinPRO, including on the mixture-of-experts model. However, the strongest average score did not mean winning every individual benchmark or every sampling budget.

With a four-step delay on the 8B model, MinPRO averaged 58.3%, versus 56.9% for M2PO. Alternative prefix statistics—arithmetic mean, geometric mean and cumulative product—scored lower in that experiment. MinPRO also retained averages of 57.3% and 57.2% at delays of eight and sixteen steps, respectively.

What remains unresolved

The study does not establish performance on coding, broader language tasks or models exceeding 100 billion parameters. The reported comparisons do not provide uncertainty estimates establishing whether small score differences are statistically reliable.

Appendix diagnostics also show stronger attenuation of later tokens. The authors explicitly caution that these aggregate weights cannot establish whether useful exploratory reasoning is preserved. MinPRO therefore offers empirical evidence for a practical weighting strategy, rather than proof that its proposed mechanism explains every gain or that it resolves stale-data training generally.