Training an AI Draft Model to Disagree at Crucial Moments

Speculative decoding normally rewards agreement. A small draft model proposes several tokens, and a larger target model checks them together. Accepted proposals accelerate generation; a rejection triggers replacement sampling. In the idealized mathematical setting, this procedure preserves the target’s next-token probability distribution.

A preprint by Hongyin Luo and colleagues, Adversarial Speculative Reasoning Improves Long Horizon Agents, asks whether agreement is always desirable for agents working through lengthy software tasks. Its central proposal is to train the draft to invite rejection at important reasoning decisions—not to change the target’s weights, but to change how those decisions are computed and stored.

Why position inside a block might matter

The researchers studied GLM-5.3-NVFP4 with a DFlash draft and DeepSeek-V4.1-Flash with a DSPARK draft. Their verification blocks contained eight and six positions, respectively. These serving systems use numerical approximations, so repeated computations need not produce identical probabilities.

To isolate variation within a verification pass, the team repeated computations while holding the input tokens and stored context fixed. They compared token log-probabilities across repetitions at different positions within a block.

For both models, average numerical variation increased by roughly 4% of the first position’s level per additional position. On GLM-5.3, the last position was 34% noisier than the first. Large deviations also became more frequent. The first position, meanwhile, had approximately the same variation as ordinary single-token decoding.

The authors attribute this pattern to how attention processes a block. Later positions depend on earlier key-value representations computed during the same pass, inheriting their numerical variation. The first position instead attends to previously cached context.

A separate experiment followed identical text written into the key-value cache through either blockwise verification or single-token decoding. Across 96 GLM-5.3 tasks, the researchers found an excess of extreme probability deviations emerging after about 30 assistant turns. They interpret the measurements as roughly a 10–20% excess, although estimates varied and some confidence intervals included no difference. Smaller deviations were indistinguishable, and the extreme-tail excess did not persist in every later interval.

Making rejection useful

The proposed method targets moments when an agent commits to a decision: interpreting an ambiguous requirement, choosing an implementation approach, or deciding whether to investigate an unexpected result.

The team constructed paired continuations sharing the same preceding context. For GLM-5.3, training contrasted decisions framed around specifications or observed evidence with decisions framed around likelihood, risk, or simplicity. For DeepSeek, it contrasted continuations that ran a test or script with those that moved on without doing so.

The draft was trained in the apparently wrong direction: toward the undesirable continuation. The intended effect is disagreement precisely where the target produces a preferred decision.

Under the unchanged rejection-sampling rule, a rejected proposal is replaced using a residual distribution. That replacement becomes the first position of the next verification block, where its key-value representation is computed under the least noisy conditions measured in the study.

This does not force the target to choose a better answer at that instant. The authors’ hypothesis is that preferred decisions become more durable over a long trajectory when encoded more precisely. The mathematical distribution-preservation guarantee assumes a fixed target distribution; the paper’s practical argument concerns numerical differences in the served implementation and their downstream effects.

Coding results and their limits

Each benchmark configuration was evaluated in five independent runs. On DeepSWE, which grades repository-level tasks using hidden tests, GLM-5.3’s reported success rate rose from 60.6% to 67.6%. The seven-percentage-point improvement had a reported 95% confidence interval of 2.6 to 11.4 points.

For DeepSeek on the same benchmark, the main text reports an increase from 67.8% to 71.8%. Its confidence interval for the gain, −1.3 to 7.3 points, included zero. On NL2Repo-Bench, where agents build Python libraries from requirements, DeepSeek’s mean test pass rate increased from 65.0% to 66.4%, with a reported interval of 0.4 to 2.4 points for the improvement.

Training drafts toward the desirable behavior instead reduced DeepSWE performance for both targets. Other draft-training preferences did not produce comparable improvements for DeepSeek. These comparisons support the proposed explanation, but the claimed durability of particular decisions remains a hypothesis.

The updated parameters represented 0.28% and 0.06% of the respective target-plus-draft systems. Speed costs were assessed through token-agreement measurements and a greedy-chain approximation: estimated accepted draft tokens were unchanged for GLM and fell about 1.8% for DeepSeek. That is narrower evidence than a direct end-to-end latency measurement.

The preprint suggests that speculative decoding may influence more than throughput in numerically approximate serving systems. Its evidence comes from two target-and-draft configurations on coding benchmarks, with uneven statistical strength across the reported gains—not a demonstration that deliberate disagreement universally improves AI reasoning.