Hybrid Spark pairs prediction with copying to speed up AI code editing

Two ways to anticipate the next tokens

In their November 3, 2026 arXiv preprint, Cohere researchers Ekagra Ranjan and Bharat Venkitesh describe Hybrid Spark, a framework for combining two approaches to speculative decoding. The supplied version is a preprint; its reported results should not be treated as independently validated findings.

Speculative decoding lets a cheaper process propose several tokens—pieces of text—for a larger target model to check together. Accepted proposals allow generation to advance farther in one verification pass. The benefit depends on whether those accepted tokens outweigh the extra drafting and checking work.

The paper combines DSpark, a neural drafter that proposes seven tokens, with an n-gram drafter that searches the request’s existing tokens for repeated sequences and copies a continuation of up to 15 tokens. Prediction can handle new material; copying can cheaply reproduce unchanged stretches of a program. Neither consistently wins across all content.

Giving copying a confidence score

A central difficulty is that the copying process does not naturally report how likely its proposals are to be accepted. Without comparable confidence estimates, a serving system cannot easily decide whether checking a copied token is a better use of computation than checking another request’s prediction.

The authors address this with logistic regression, a small statistical estimator. It uses features such as match length, whether competing matches suggest the same next token, and how far a copied passage extends backward. It estimates acceptance separately along the proposed sequence, including distinct treatment of its first token.

The estimator was fitted separately for each target model using generated outputs from 764 general-text SPEED-Bench prompts. Another 766 prompts were held out, although held-out loss informed feature selection. None of the editing benchmarks was used for fitting or feature selection. Limited cross-model tests also found little throughput change when coefficients were borrowed from another model.

Hybrid Spark then assigns verification capacity a shared “price”: the expected benefit that capacity could provide elsewhere in the batch. Each request keeps the candidate whose estimated useful tokens justify that price. When the copying candidate contains the entire neural draft as its prefix, a separate rule retains the longer candidate.

Token-dependent selection and allocation run on the GPU. The CPU still sizes the verification budget using older estimates, preserving asynchronous scheduling rather than waiting for a fresh GPU result mid-step.

What the benchmarks showed

The evaluation used vLLM on NVIDIA H100 80 GB GPUs, with client concurrency set to batch sizes of four, 32 or 128. Comparisons used matching GPU layouts, asynchronous scheduling and disabled prefix caching. Most tests used temperature zero, making target generation deterministic.

Across five editing workloads and the three models used for the headline comparison—Qwen3-4B, Qwen3-8B and Gemma 4 12B—the authors report output throughput of 1.10–1.62 times DSpark’s at batch size four and 1.04–1.27 times at batch size 128. These are improvements over an already accelerated baseline, not over ordinary decoding.

The advantage generally declined as edits became more extensive. On NextCoder, light edits with 15–20% edit distance gained approximately 28–33% at batch size four; heavy edits with 60–70% distance gained about 3–12%.

Chat provided little benefit. On MT-Bench at temperature zero, Hybrid Spark delivered 0.97–1.01 times DSpark’s throughput: between roughly 3% slower and 1% faster. InstructCoder also showed small regressions at the largest batch size. Running both drafters is therefore not a universal improvement.

The boundaries matter

The measurements concern total generated tokens per benchmark duration, not a demonstrated improvement in code correctness or a comprehensive account of user-facing latency. Most results average three fresh-server runs, with bootstrap confidence intervals. Some editing datasets contain only 19–30 distinct examples and are repeatedly sampled to sustain the requested concurrency.

The headline EditBench results exclude requests exceeding 6,000 context tokens. Much larger gains elsewhere in the paper occur where DSpark’s drafter performs poorly; the authors explicitly caution that stronger drafters would likely reduce those advantages.

There is also a distribution-preservation condition. The authors argue that the main selection policy preserves the target model’s output distribution at any target temperature when drafting is greedy. With probabilistic drafting and a nonzero target temperature, that policy can introduce bias; restricted alternatives are required.

The practical finding is consequently narrower than a general AI speedup: content-aware copying can improve an existing speculative-decoding system on repetition-heavy editing workloads. Benefits for other hardware, live mixed-traffic deployments, alternative drafters and reinforcement-learning rollouts remain unestablished by these experiments.