HybridSpec speeds hybrid language models by checking fewer draft tokens

Why checking more can mean doing worse

In the November 2, 2026 preprint HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN–MoE Models, Haiyu Wang, Sai Qian Zhang, Zechun Liu and Steven Li describe a system for reducing the cost of checking proposed text. The paper, arXiv:2611.04055v1, reports results from researchers affiliated with Meta AI and New York University.

The approach builds on speculative decoding. A smaller drafter proposes several tokens—pieces of text—and a target model checks them together. Under greedy decoding, which chooses the highest-scoring next token, the system accepts the matching prefix and uses the target’s choice at the first mismatch. Computation spent checking later proposals is then wasted.

Fast parallel drafters such as DFlash make this verification stage especially important. HybridSpec focuses on targets combining Gated DeltaNet, or GDN, layers with mixture-of-experts, or MoE, layers. GDN maintains a fixed-size recurrent memory; MoE routes tokens to selected expert networks rather than using every expert.

In the authors’ single-request profile of Qwen3.6-35B-A3B, increasing verification length from four to sixteen target-input tokens raised target GPU time by about 71%. Almost all the increase came from GDN and MoE work. A separate dense Qwen3-4B comparison showed increases of roughly 4%–6%. These are setup-specific measurements, not universal architectural constants; the hybrid profile also used separate decoding runs for each length.

Choosing the work before doing it

HybridSpec first estimates how likely successive draft tokens are to be accepted, using calibrated drafter confidence and entropy signals. It combines those estimates with measured target latency to select a verification length on the GPU. Its objective is total emitted tokens divided by total decoding time, rather than simply maximizing tokens accepted in each round.

The full proposal contains fifteen draft tokens. Verification length also counts an anchor: the newest emitted token that the target has not yet processed. Vanilla DFlash therefore checks sixteen input tokens each round, while HybridSpec usually selects fewer.

A second decision happens inside the target. HybridSpec initially processes the selected input length plus, when allowed, one extra token through the first GDN layer and the first MoE router. It pauses before any routed expert executes.

The router reveals whether that extra token would reuse experts already needed or activate new ones. HybridSpec uses this observation to estimate additional expert costs across the model, keeping the token only when its expected acceptance benefit justifies the predicted remaining latency. Later-layer costs remain estimates: the system does not observe every layer’s routing before deciding.

GPU-controlled CUDA Graphs then resume verification for the retained tokens without repeating the initial computation or copying the selected length to the CPU.

Rebuilding memory more efficiently

The system also changes GDN verification. The default SGLang implementation examined in the paper updates recurrent memory sequentially and stores a full state after every token, allowing restoration after rejected proposals.

HybridSpec instead processes the short token block in parallel and retains compact mathematical factors. After acceptance, one GPU kernel reconstructs the committed states across all GDN layers. This reconstruction is parallel across layers; the model’s forward pass remains sequential through its layers.

In isolated benchmarks covering thirty GDN layers, verification plus state reconstruction was 1.54–4.04 times faster than the default kernels at lengths four through sixteen when every draft token was accepted. These figures exclude input-projection production and the two-stage verification machinery. They are component speedups, not whole-model acceleration.

What the evaluation found

The main evaluation used SGLang 0.5.16, NVIDIA A100 80 GB GPUs, and three Qwen targets: two 35B models and a GPTQ-Int4 122B model. The larger target used two GPUs without NVLink. Researchers separated 280 calibration prompts from 280 test prompts across seven SPEED-Bench categories.

For single-request greedy decoding, HybridSpec’s category-average speedups over autoregressive generation were 1.88, 1.83 and 1.75 times. The reported advantage over vanilla DFlash was 1.30–1.33 times. EVICT, the strongest competing baseline in these comparisons, reached category-average speedups of 1.51–1.65 times over autoregressive decoding.

These metrics measure post-prefill decoding throughput, not complete request latency. HybridSpec accepted slightly fewer tokens per round than DFlash but completed rounds faster. Temperature-one sampling also showed gains on the two 35B targets.

Where the claims stop

The authors prove output preservation in exact arithmetic under stated assumptions, including causal prefix invariance and exact state recovery. Sampling preservation specifically relies on their point-mass acceptance rule and choosing verification length before verification draws. Floating-point implementations can differ: HybridSpec and DFlash did not produce identical outputs in the reported math evaluation.

On 1,319 GSM8K problems, their accuracies were 96.06% and 96.21%, respectively, meeting the authors’ prespecified one-percentage-point noninferiority criterion. That supports a narrow task-quality comparison, not universal equivalence.

Scaling also matters. With four concurrent clients on Qwen3.6-35B-A3B, the batched version reached 497 tokens per second versus DFlash’s 451—a smaller 1.10-times advantage. The authors say code will be released upon acceptance. The preprint’s central result is therefore a measured efficiency improvement under specified conditions, not a guarantee for every serving system.