Faster AI text generation by keeping possible continuations apart

How can a language model generate text faster without changing the distribution of answers it produces? One approach, speculative decoding, lets a cheaper drafting model propose upcoming tokens—pieces of words or punctuation—which the main model checks in parallel. The challenge is making those guesses useful.

In a preprint submitted on November 9, 2026, Haoyu Huang and colleagues describe Multi-Future Drafting, or MFD. Instead of blending several possible continuations into one set of predictions, their drafter learns to represent alternatives separately. The reported advantage is chiefly for generating responses with few requests running simultaneously.

When reasonable guesses make an unreasonable sequence

Some existing drafters predict a whole block of future tokens in one computational pass. A tree-building procedure then assembles candidate continuations, allowing the main model to check several branches together. Each node represents a candidate token extending a particular prefix: the beginning of a continuation.

The problem is how those prefixes are ranked. Multiplying separate probabilities for each token position treats choices at different positions as independent. But wording is connected: choosing one opening changes which words should follow.

The paper illustrates this with two continuations after “Let’s break”: “this down step by step” and “down the problem step by step.” Mixing their individually likely tokens can instead produce “this the problem step step.” The authors call such spliced prefixes chimeras.

Their mathematical analysis shows a structural limitation. Excluding tied scores, a product-ranked tree containing two mutually exclusive continuations that differ at m positions must also contain at least 2^(m−1)−1 spliced prefixes. Better training alone cannot remove this representational problem.

Real model outputs are less clear-cut than that idealized example. Across 628 benchmark prompts on two Qwen models, the researchers classified roughly 27–33% of nodes in 64-node single-mode trees as chimeras. Their detector looked for tokens very unlikely after their own prefix but plausible after another prefix in the same tree—not proof that the target could never produce them.

Several alternatives, one verification tree

MFD adds learned numerical vectors, called mode embeddings, to the drafter’s input. Each vector conditions a different version of the proposed block. The experiments generally use eight modes, processed side by side in one drafting pass while sharing the model’s weights and context information.

Training encourages specialization. For each training block, the mode that best covers the target continuation receives most of the learning update; the others receive a small share. The main comparison matches drafting work per training step by using fewer starting positions for the eight-mode model.

At generation time, the system combines candidates into one tree. It ranks each complete prefix by its strongest mode-specific score, rather than switching modes independently at every token. Shared prefixes occupy only one node. The timed experiments give modes equal prior weight.

The main model then follows its own token choices through that tree. It does not have to select the right mode beforehand. Indeed, experiments that committed to one predicted mode lost the benefit. Improvements also persisted with deterministic generation: even when the target’s answer is fixed, the smaller drafter remains uncertain about it.

MFD reduced chimeras rather than eliminating them. They still occupied approximately 22–24% of 64-node mixture trees.

What the speed measurements mean

The researchers tested four Qwen targets with 4–27 billion parameters on mathematics, programming and conversation benchmarks. Main timings used one NVIDIA RTX PRO 6000 GPU, one request at a time, and excluded initial prompt processing.

With fine-tuning on approximately 100,000 prompts, MFD was about 5–8% faster than the strongest comparison method across the reported model-and-temperature averages. With a larger 1.42-million-prompt fine-tuning set, its highest five-task average reached 7.78 times ordinary token-by-token decoding on Qwen3.5-4B. That larger multiplier compares against ordinary decoding, not against an already accelerated competitor.

In a separate SGLang serving implementation, MFD was 7–9% faster than an equally fine-tuned single-mode tree at batch size one. Adding modes requires few extra parameters, but it still adds computation.

Where the advantage stops

As simultaneous requests increased, that computation became a liability. In the serving tests, MFD fell behind a tree using the released single-mode drafter from batch size eight. A separately trained variant switched to a single-mode path at larger batches, avoiding the slowdown but no longer using mixture drafting there.

The evidence is also bounded by one GPU type, selected benchmarks and tree budgets chosen using the timed prompts. Hybrid-model results were measured only in the experimental harness. Verification preserves the target distribution mathematically, while reduced-precision arithmetic can change choices at near-ties.

The preprint therefore supports a specific possibility, not a universal acceleration claim: retaining coherent alternatives can make parallel guessing more efficient when response latency, rather than large-batch throughput, is the priority.