A shared shortlist could make long-context AI faster

Why speed up the search?

Attention lets a language model use information from earlier parts of a prompt. In sparse attention, the model selects a limited set of earlier tokens rather than attending to the entire context. DeepSeek Sparse Attention (DSA), the method studied here, uses an indexer to score the visible history and select the highest-scoring tokens.

That selection can become a bottleneck of its own: the indexer scans the causal prefix—the tokens available so far—for each query. The paper reports that at a 128K-token context, indexing accounts for 54.2% of prefill GPU time in its measurement. Prefill is the stage when a model processes the input prompt before generating its response. The researchers ask whether repeated searches can be reduced without substantially changing which tokens the model uses.

A shared bank, then individual choices

ADIndex groups nearby queries. A lightweight, group-level indexer scans the history once for a group and selects a shared bank of candidate tokens. The original DSA indexer then scores candidates separately for each query, adding tokens that became visible after the group’s first query. This refinement keeps the final selection query-specific; the method does not simply give every query the same attention pattern.

The researchers first calibrate the group size and bank budget separately for each model layer. Their motivation is that overlap between neighboring queries’ selected tokens varies across layers: some layers can share a smaller bank, while others need more capacity. They then train the group indexer through distillation, using the original indexer’s scores as a teacher. The backbone model and original indexer remain frozen during this training.

Tests used DeepSeek-V3.2 and GLM-5.1, with the method implemented in SGLang and evaluated on NVIDIA B300 GPUs. The researchers compared results with the original DSA and several indexer-acceleration methods. Task evaluations used LongBench, LongBench v2 and RULER. The group indexer was trained on sampled long sequences, while a separate set of 100 sequences was used for layer-wise calibration.

What the tests found

On the reported benchmarks, ADIndex scores were generally close to those of the original DSA. For example, on RULER at 128K tokens, the scores were 93.34 versus 93.69 for DeepSeek-V3.2, and 94.49 versus 94.97 for GLM-5.1. These results support the authors’ description of performance as “near-lossless” in these tests, not a claim that the method never changes outputs or preserves every capability.

At a 256K context, the paper reports prefill indexing-kernel speedups of up to 6.22 times and MTP target-verification indexing-kernel speedups of up to 3.40 times, depending on the reuse configuration. MTP target verification is a stage in which the system checks proposed tokens; this result is not a speedup for ordinary autoregressive decoding. For end-to-end serving, the paper reports a 1.54-times speedup at a 128K KV-cache length. Kernel speedups describe particular operations and should not be read as equivalent improvements in total user-facing response time.

Context length matters. The authors report that ADIndex can be slower than DSA at shorter contexts, where building a shared bank may cost more than it saves. The method falls back to the original indexer in some short-context cases, but the paper’s end-to-end measurements still show a slowdown at 16K in one reported setting. Gains grow as the history becomes longer and repeated full-prefix scans become more expensive.

What remains uncertain

The results come from two models, a specific software implementation and B300 GPUs. Performance on other models, hardware, workloads or serving systems is not established by these experiments. Layer settings are calibrated using a limited sample of sequences; how well those choices transfer to very different inputs is an open practical question. The reported efficiency also depends on implementation details, and the paper notes that simplified counts of scoring operations do not include every overhead that affects real latency.

ADIndex is presented in an arXiv preprint, which is not by itself evidence of peer review. The work offers a targeted engineering approach to one long-context bottleneck, with promising results in the tested setup—not proof that all long-context language models will become faster without trade-offs.