Teaching diffusion models to do more with fewer decoding steps

A different training signal for parallel generation

Generating several tokens at once offers an alternative to left-to-right text generation. Discrete diffusion models do this through iterative denoising, but obtaining good answers can still require many model evaluations. Distillation aims to transfer a teacher model’s capabilities to a student that can operate with fewer evaluations or lower computational cost.

In a preprint posted on November 3, 2026, Peng Tu and colleagues introduce Bidirectional Centered Score Matching, or BCSM. Their approach changes how the student learns the teacher’s preferences among possible tokens at each output position. Experiments cover both language models and a vision-language model.

The distinction matters: BCSM is a different within-position training objective, not a mechanism that explicitly reconstructs the joint relationships between simultaneously generated tokens.

Matching relative preferences

A common distillation objective uses Kullback–Leibler divergence, or KL, to match teacher and student token probabilities. The paper explicitly acknowledges that token-wise KL exactly matches the corresponding product-form block distribution. Its limitation is not mathematical failure to match that distribution: dependencies excluded by the parallel output factorization remain excluded.

The authors instead investigate whether another optimization geometry provides more useful supervision when inference steps are aggressively reduced. They also address candidate selection. Restricting training to tokens preferred by the teacher can omit tokens that the student currently favors, leaving some student mistakes outside the selected training support.

BCSM builds an adaptive candidate set from both models. It examines their highest-ranked logits—the unnormalized scores used to calculate token probabilities—and locates a boundary between prominent candidates and the flatter tail using a knee-detection criterion. The union of the teacher’s and student’s selected tokens becomes the active set.

Within that set, BCSM matches relative logit differences. Rather than comparing every pair explicitly, it subtracts a weighted mean from each model’s logits and aligns the resulting centered scores. Adding the same constant to every logit does not change token probabilities, and centering removes this irrelevant offset.

A weighted variance identity makes the centered calculation exactly equivalent to the proposed active-set pairwise squared log-ratio objective. This reduces that matching calculation from quadratic to linear cost in the active-set size. It does not make the entire training process linear in that size: the method still computes full-vocabulary student probabilities and retrieves high-scoring candidates.

The implementation injects detached corrective directions into the student’s gradients. An optional transition-aware version also aligns how centered scores change between two noise levels sampled along the same forward diffusion trajectory.

What the experiments report

On HumanEval, the static method scores 42.1 with LLaDA-8B at an average of 42.9 function evaluations per block, compared with 36.0 for the undistilled 256-evaluation reference. That corresponds to approximately six times fewer evaluations and a 16.9% relative score increase—not a demonstrated sixfold reduction in elapsed time.

For the Bard-4B-VL experiments, static BCSM reaches 35.67 on MMMU-Pro, labeled MMMU in the paper, at single-step inference. The undistilled student scores 30.98 at the same budget, giving the reported 15.1% relative improvement. Its 32-step reference scores 32.14. Across the five benchmarks in the main vision-language table, a BCSM variant achieves the highest reported average at each tested inference budget.

The transition-aware extension is not uniformly better than static BCSM. In an OCR ablation, it leaves the single-step score unchanged at 40.9, while improving the eight-step score from 65.9 to 66.8 and the sixteen-step score from 71.3 to 71.9. Larger transition weights eventually reduce performance, indicating that this additional signal needs balancing.

Better agreement is not a guarantee of coherence

On 20,252 image-captioning validation samples, BCSM has lower non-reference n-gram rates than the tested KL baselines. This metric measures disagreement with reference captions, not whether an output is inherently invalid. Qualitative OCR examples likewise show improved reference recovery, alongside failures involving truncation and character deviations.

Some appendix passages make stronger claims about eliminating inconsistent generations. Those claims should not be read as guarantees: the main text explicitly states that BCSM shares the cross-position representational limitation of other position-wise objectives and still struggles with exact long-sequence recovery.

The preprint therefore supports a narrower conclusion: in the reported settings, changing candidate coverage and relative-logit supervision improves useful quality–evaluation trade-offs without establishing explicit cross-position dependency recovery.