LeapQuant trims the memory burden of linear-attention models

The bottleneck behind a compact memory

In conventional attention, a model can refer back to stored representations of earlier tokens. Linear-attention layers instead carry information forward in a fixed-size recurrent state. This can make processing long sequences more manageable, but it does not make the state free: during generation, a layer repeatedly reads and updates it. The authors argue that moving this state between GPU memory and processing units can become a bottleneck, and that keeping many cached states also consumes memory.

A straightforward fix—store the state with fewer bits—can introduce rounding errors. Because each new update starts from the previously rounded state, errors can build up over a long generation. The state also contains a few unusually large values, or outliers. Those can set a wide scale for quantization, leaving smaller values represented less precisely.

Quantize less often, protect the outliers

LeapQuant combines three measures. First, it quantizes the recurrent state at the end of a window of tokens rather than after every token. Within the window, it keeps the boundary state fixed and buffers higher-precision updates. Outputs are calculated using that state and the buffered updates; the full state is reconstructed and quantized at the next boundary. This reduces how often rounding enters the recurrent process, though it requires temporary storage for the updates.

Second, the method represents some of the state’s largest patterns separately as a small number of high-precision “Compensator Tokens.” These are mathematical components of the state, not actual words generated by the model. Keeping them precise leaves a residual—the rest of the state—that is easier to quantize. Third, LeapQuant smooths that residual across channels before quantization to make its values more evenly scaled. The method is training-free and, according to the authors, requires no calibration data.

What the tests report

The researchers evaluated the method on five hybrid language models from the Qwen, Kimi and GLM families, using tasks that included mathematics, science questions and code generation. Their implementation used a 16-token window and a small set of compensator components in its main configuration. They compared model-task accuracy with an FP32-state reference and tested performance on three NVIDIA GPU types.

At 8-bit state precision, the paper reports accuracy comparable to the FP32 reference across 12 model-task pairs: LeapQuant’s average was 84.4%, compared with 84.0% for FP32, and every pair was within 2.5 percentage points. In one reported long-reasoning example, ordinary per-token FP8 and INT8 quantization performed poorly, while LeapQuant preserved the reference result. These are benchmark outcomes, not evidence that every model or real-world task will behave similarly.

The authors report a 3.4-fold reduction in recurrent-state memory traffic and reductions of 41% to 56% in total memory use for three tested models with prefix caching. In end-to-end inference tests, throughput gains varied by model, task and GPU; on the B200, the reported range was 1.19 to 2.98 times. A single average speedup can conceal this variation. The paper also reports larger speedups for individual attention-layer kernels than for full-model inference, where other operations remain part of the workload.

What remains uncertain

These findings come from a preprint and experiments conducted by its authors; they should not be treated as independent confirmation. The tests cover selected models, tasks, implementation choices and GPUs, and the paper does not establish that the same gains will hold across other hardware, serving systems or workloads. Accuracy on benchmark tasks is also only one measure of model quality. The authors’ results at 4-bit precision are more mixed across models than the 8-bit findings, underscoring that stronger compression can carry greater risk.

The central claim is therefore specific: for the tested hybrid models and setups, delaying quantization and preserving outlier structure helped reduce memory costs while maintaining benchmark accuracy at 8 bits. Whether that translates into a dependable improvement in broader deployments remains to be tested.