A Subgroup-Aware Baseline Offers Modest Gains in AI Math Training

A correct answer can emerge from a productive strategy or from an approach that usually fails. Under a shared group baseline, those successes receive identical training credit if their rewards match. The preprint “One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets in GRPO”, dated 31 October 2026, asks whether accounting for strategy-based differences can improve mathematical reasoning. Its results support modest benefits under a matched experimental setup, rather than an across-the-board improvement over ordinary GRPO.

What the shared average leaves out

Group Relative Policy Optimization, or GRPO, samples several attempted solutions for a problem and scores their answers. It estimates each attempt’s advantage—its relative training credit—by subtracting the group’s average reward and dividing by the group’s reward standard deviation. Equally rewarded attempts within a group therefore receive equal advantages. When every reward is identical, the researchers set all advantages to zero.

Zhou and colleagues argue that this shared reference overlooks structure inside the group. Attempts following one strategy may succeed frequently, while attempts following another rarely do. A single average does not explicitly distinguish performance within a strategy from that strategy’s performance against alternatives.

The researchers construct subgroups using short strategy previews generated by the model. Each problem has four strategy slots, with eight attempted solutions per slot: 32 solutions overall. Extracted, usable previews condition solution generation. If too few are available, remaining slots receive empty previews and their solutions use the plain problem prompt. The previews condition generation and define subgroup membership; they are not themselves rewarded or optimized.

Two directions for assigning credit

The adjustment starts with a subgroup offset: the subgroup’s average reward minus the overall group average. A signed coefficient, λ, determines how much of this offset is added to the baseline.

For λ between zero and one, the baseline moves toward the subgroup average. The authors call the positive direction Alignment. Credit combines comparison against the entire group with comparison against the attempt’s own subgroup. At λ = 1, the baseline becomes the subgroup mean, although normalization still uses the global standard deviation.

With negative λ, the baseline moves in the opposite direction. This Contrast adjustment raises advantages for attempts in above-average subgroups and lowers them for attempts in below-average ones. It emphasizes differences between strategies rather than discounting those differences.

Neither direction changes the answer-checking reward or the form of the optimization objective. The intervention changes the baseline used to calculate advantages. Setting λ to zero restores the shared group-mean baseline.

The comparison that matters

The authors train Qwen3-8B-Base and Olmo3-7B-Base on DAPO-Math-17K, then evaluate on AIME 2024, 2025 and 2026, plus HMMT 2025. Evaluation queries models directly, without strategy previews.

They report pass@1, mean accuracy across sampled answers, and pass@16, whether at least one of 16 attempts solves a problem. Each benchmark contains 30 problems. Each problem receives 32 repeated evaluations with 16 samples apiece, and results are averaged across repetitions.

The key control is Strategy-GRPO, which uses the same strategy-conditioned sampling but no baseline adjustment. This comparison isolates the adjustment from the effects of introducing strategies. The main settings use λ = +0.3 for Alignment and −0.3 for Contrast.

Against that control, both directions improve pass@16 on all four benchmarks for both models. Contrast generally delivers larger gains. For Qwen, its AIME 2026 pass@16 rises from 53.8% to 58.8%; for Olmo, the corresponding result rises from 62.2% to 65.2%. Single-attempt accuracy is less consistently improved: Olmo’s HMMT pass@1 falls from 16.3% to 14.6% under Contrast.

Ordinary GRPO remains an important reference. Strategy-GRPO trails it or ties it throughout the reported results, which the authors attribute to a mismatch between strategy-conditioned training and direct evaluation. Contrast exceeds ordinary GRPO’s pass@16 on all four Qwen benchmarks. On Olmo, however, its pass@1 remains below ordinary GRPO on every benchmark. Gains against the matched control do not establish universal superiority over ordinary GRPO.

Useful structure, bounded evidence

Shuffling offsets between subgroups pushes both adjustments below the matched control on all four Qwen benchmarks in pass@16. The authors interpret this as evidence that subgroup–offset correspondence matters, while acknowledging that the experiment cannot separate this explanation from the possibility that miscalibrated offsets generally hurt.

The coefficient sweep also cautions against stronger adjustments: −0.3 outperforms −0.5 and −0.7 on average under Contrast. That sweep covers only Qwen.

Strategy generation adds sampling cost, evaluation is confined to mathematics, and the method supplies no automatic rule for choosing the coefficient’s sign or magnitude. The supported takeaway is bounded: strategy-aware baselines can improve multi-attempt coverage in these experiments, with Contrast the stronger direction overall.

Source note: The arXiv citation metadata titles this preprint “One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets”; the PDF title adds “in GRPO”. The sources do not explain this difference.