BitDecoding-FP4 aims to shrink AI memory without sacrificing answer quality

Why long context becomes expensive

As a language model generates an answer, it stores numerical information about earlier tokens—pieces of text—in a key-value, or KV, cache. Attention repeatedly consults this cache to decide which earlier information matters. The cache grows with both conversation length and the number of requests being served, consuming GPU memory and memory bandwidth.

Four-bit floating-point formats, known as FP4, offer compact storage. NVIDIA’s recent Tensor Cores can also multiply these compressed numbers directly. But compression introduces rounding errors, and faster multiplication does not automatically accelerate the surrounding computation.

BitDecoding-FP4, developed by researchers affiliated with the University of Edinburgh and Tencent Hunyuan, addresses both problems together. It targets the decoding stage, when a model generates successive tokens, rather than making the entire model four-bit.

Three ways to protect accuracy

The method uses microscaling: small groups of four-bit numbers share a scale that adjusts their numerical range. The paper examines two formats, NVFP4 and MXFP4, with different group sizes and scale representations.

First, the researchers improve the query, the numerical representation used to look for relevant cached information. Rounding a query affects its comparisons with many previous tokens. Their solution stores an ordinary FP4 approximation plus a second FP4 approximation of the error left behind. In the reported NVFP4 analysis, the query’s share of attention-score error falls from about 60% to about 1%. This is not a reduction of overall answer error from 60% to 1%.

Second, they address very small attention probabilities. In long contexts, many probabilities are tiny, and NVFP4’s scale format can round their block scales to zero. The method raises probabilities before quantization and changes the scaling rule for small blocks. The common multiplier cancels during normalization. These changes lower the underflow threshold; they do not eliminate every possible underflow.

Third, they reduce unusually large key components that would otherwise dominate a shared scale. They subtract a calibrated mean, apply a mathematical rotation that spreads extreme components, and choose between two candidate scales. Calibration uses eight short prompts and model-generated continuations arranged across the context window.

Making compression translate into speed

The system also reorganizes softmax, the calculation that converts attention scores into weights. With relatively few query heads, a conventional layout leaves many GPU threads idle. The researchers transpose the calculation so threads work across cached tokens, making reductions more parallel and combining probability quantization with that work.

They also place the second query approximation into otherwise unused matrix-multiplication space, avoiding a separate multiplication instruction for each cache block. The implementation integrates with the vLLM serving engine’s paged memory management.

Importantly, this is a mixed-precision system. It retains the first 16 tokens and the incomplete page of recent tokens in BF16, a 16-bit format, while older full pages use FP4.

What the experiments found

On two Qwen3 models, the NVFP4 version scored 67.27 versus 67.69, and 72.74 versus 73.15, for BF16 caches when averaged across seven benchmarks. These covered long-context tasks, mathematics, coding and knowledge questions. MXFP4 remained within one percentage point of the BF16 average. Both variants exceeded the tested FP8-cache baseline’s average score, although individual task results varied.

For end-to-end serving on NVIDIA B300 GPUs, the authors report 2.4–3.0 times the peak throughput of BF16 caches with 64K- and 128K-token inputs and 2K-token outputs. The 128K setup used two GPUs. Memory capacity contributed to the gains: the compressed cache accommodated more simultaneous requests before queuing or preemption became necessary.

Separately, attention-kernel tests reached speedups of 2.99 times on B300 and 4.01 times on RTX PRO 5000. Those figures concern one computational component, not the complete serving pipeline. In reinforcement-learning experiments on the two Qwen3 models, validation accuracy with the proposed rollout cache closely followed BF16.

Boundaries and preprint status

Larger-model experiments also reported substantial cache savings: from 71.88 to 20.27 GiB for GLM-4.7 at 200K tokens, and from 90.46 to 34.41 GiB for Hy4-preview at one million tokens. But Hy4-preview uses a different attention architecture. Its implementation retains positional information in BF16 and reconstructs compressed latent blocks for an FP16 weighted calculation; not both attention products run natively in FP4. The authors omit its latency speedup because its sparse attention examines a fixed, limited set of tokens per step.

The small average score gaps do not establish equal performance on every task. Hardware and workload choices matter, and comparisons have qualifications: TurboQuant’s mixed-precision setup was disabled, while SAW-INT4 latency was omitted because it used a different serving engine.

The supplied arXiv record identifies this as version 1, submitted on 6 November 2026. It does not establish peer-review acceptance. The authors provide an implementation, but the performance figures remain results reported in this preprint, not guarantees for other deployments.