ChainFlow drafts several AI tokens by predicting where a model’s internal states go next

Large language models usually generate text one token at a time. Speculative decoding tries to reduce that sequential work: a smaller drafter proposes a continuation, and the larger target model checks several proposed tokens in one pass. The useful speedup depends on two competing costs: how quickly the drafter works, and how much of its proposal survives verification.

In an arXiv preprint, Selim Aktas and colleagues introduce ChainFlow, which drafts by predicting the target model’s future internal representations. Their experiments show gains over two competing drafters under a specific low-load serving setup, alongside substantial limits at higher concurrency.

Starting from the model’s recent trajectory

A hidden state is a numerical representation produced inside a model as it processes tokens. ChainFlow takes the target’s recent final-layer hidden states and compresses them with a variational autoencoder—a neural network trained to encode information into a smaller representation and reconstruct it.

Rather than starting its prediction from random noise, the drafter extends the direction between the last two compressed states. That gives it an initial estimate for eight future positions.

A network trained with flow matching then adjusts those estimates toward the future states seen during training. Here, flow matching means learning how to move an initial representation toward a desired one. The drafter is trained using two numerical update steps, but the authors find that a single step gives the highest throughput when serving requests.

The predicted states are decoded and converted into token scores using the target model’s frozen output head. Lightweight feedback components then adjust those scores according to tokens already chosen within the draft. This matters because proposing several positions together does not, by itself, make their choices consistent with one another.

The branching check supplies the main gain

ChainFlow can emit a single chain or a tree of possible continuations. Its headline configuration keeps eight candidate nodes at each of four depths, producing 32 draft nodes for the target to check. Each node is evaluated with its own ancestral path, rather than with unrelated branches.

The distinction is important: the simple chain was not the winning configuration. On the four-billion-parameter Qwen3.5 target, it achieved 1.23 times ordinary decoding throughput and trailed both comparison methods. The tree reached 1.87 times ordinary throughput, compared with 1.69 for DFlash and 1.35 for EAGLE-3. Those results underpin the reported advantages of approximately 11% and 39%, respectively.

The comparison used similarly sized drafters trained on the same data, one NVIDIA B300 GPU, vLLM 0.31.0, greedy decoding and one request at a time. Evaluation covered eight English domains, with outputs capped at 256 new tokens. Mathematics and code showed the largest gains; translation showed the smallest. Against ordinary decoding, pooled tree speedup reached 2.12 times on the 27-billion-parameter target.

Adaptation helps, but can erase earlier gains

The researchers also adapted English-trained drafters sequentially to Turkish, German and French. ChainFlow showed the largest new-language gains in their comparisons, although several multilingual comparisons measured acceptance offline rather than live serving throughput.

Some advantage came from vocabulary coverage: ChainFlow’s language-specific candidate shortlist covered almost all Turkish evaluation tokens, while the baselines retained an English vocabulary covering only about three in five. Adaptation also brought forgetting. Adding German removed most of ChainFlow’s Turkish improvement unless earlier Turkish examples were replayed during later training.

Faster is conditional

The tree’s extra verification work became expensive under load. On the four-billion-parameter target, ChainFlow remained faster than ordinary decoding at four concurrent requests, but fell below it at eight. At 128 concurrent requests, relative throughput was 0.45 times ordinary decoding. Both comparison methods retained gains across the tested concurrency levels.

Output preservation also needs qualification. Tree verification requires engine modifications and supports greedy decoding only. Across the reported B300 runs, 4–13% of prompts produced outputs different from plain greedy decoding. The authors attribute these differences to near-ties in half-precision calculations; the baselines also differed. These measurements therefore do not establish bitwise equivalence.

All targets belonged to Qwen3.5, and some design experiments used only a smaller 0.8-billion-parameter model. The evidence supports a promising low-concurrency drafting approach, not a universal acceleration result or an improvement in the target model’s answer quality.