FOSCAR speeds up compressed AI memory by rearranging the maths

Language models generate text one token at a time, where a token is a small unit of text. At each generation step, attention reads stored information about earlier tokens from a key–value, or KV, cache. Longer contexts and more simultaneous requests enlarge that cache, making memory capacity and data movement important constraints.

FOSCAR targets the work needed to use a compressed cache. In the preprint FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding, Donglin Zhuang and colleagues report faster attention calculations on NVIDIA H100 and B200 GPUs. Version 1, arXiv:2611.09722v1, was submitted on 6 November 2026. These are preliminary, author-reported results, not independently established performance guarantees.

Why smaller storage is not enough

Quantization represents numbers using fewer bits. Here, cached values become 2-bit or 4-bit integers, with groups sharing a scale and an offset. Reconstructing an approximate value means multiplying its integer representation by the scale and adding the offset.

That saves memory traffic, but conventional kernels—GPU routines performing the calculation—still prepare individual values for multiplication. Even when preparation happens inside attention without writing reconstructed values back to main GPU memory, it consumes instructions and on-chip resources.

The researchers identify instruction issue, the rate at which GPU schedulers can dispatch instructions, as a bottleneck. They report that peak memory bandwidth grows much faster than instruction-issue capacity from H100 to B200. Consequently, shrinking stored values can leave the processor spending proportionally more time preparing them rather than reading them.

Moving corrections instead of repeating them

FOSCAR’s central technique, affine folding, rearranges where scales and offsets enter attention’s matrix multiplications. A matrix multiplication combines rows and columns of numbers through repeated multiplication and addition.

Rather than reconstructing every cached value separately, FOSCAR applies a shared scale to the other matrix input or to a group’s partial result. It accounts for offsets through additional output corrections. The placement depends on whether quantization groups run across features within a token or across tokens within a feature.

The principle resembles factoring a common multiplier out of a sum: the arithmetic result stays the same, but fewer repeated operations are needed. For the main benchmark’s grouping and attention-head configuration, the paper calculates a sixteenfold reduction in scalar scale-and-offset operations. That is not a sixteenfold speedup for the whole kernel.

FOSCAR also constructs inputs for Tensor Cores, the GPU units specialized for matrix calculations, through bit manipulation. Separate H100 and B200 pipelines overlap data movement, input preparation and computation.

Crucially, the mathematical equivalence is to attention using the reconstructed quantized cache, in exact arithmetic. It does not restore information lost during compression, and finite-precision GPU arithmetic can introduce further differences.

What the benchmarks show

The main kernel evaluation used batch sizes of 1, 8 and 64, with context lengths spanning 8,000 to 512,000 tokens across the tested configurations. Against FlashInfer’s 16-bit BF16 baseline, geometric-mean speedups were 3.36× for FOSCAR’s 2-bit kernel and 2.83× for its 4-bit kernel on H100. On B200, they were 2.67× and 2.30×.

Comparisons with lower-precision competitors were less dramatic. Against TensorRT-LLM’s native four-bit floating-point NVFP4 kernel on B200, FOSCAR averaged 1.16× speedup at 2 bits and parity at 4 bits. These attention-kernel results should not be read as equivalent acceleration of an entire language model.

The team also integrated FOSCAR into the SGLang serving system and tested Qwen3-8B with 100,000 input tokens and 1,000 output tokens. The workloads reused 95% or 99% of each input through prefix caching.

Replacing OSCAR’s 2-bit attention kernel while retaining its quantization policy, rotations and small BF16 cache windows raised peak aggregate throughput from 269 to 487 tokens per second on H100 and from 333 to 868 on B200 at 99% cache hits. A separate configuration without those rotations or windows reached higher peaks, but should not be conflated with the accuracy-tested integration. Serving throughput reflects both calculation speed and how many requests fit in memory.

Accuracy and limits

On synthetic inputs, FOSCAR’s 2-bit output errors closely matched OSCAR’s when both used the same compressed cache. Errors relative to that cache’s reconstructed values were much smaller, supporting the distinction between compression loss and kernel arithmetic error.

Five-run evaluations on GPQA-Diamond and GSM8K found no measurable accuracy change for the OSCAR integration; the authors explicitly note that this does not establish formal equivalence. Those tests also had short median prompts, unlike the throughput workload.

Benefits are not universal: an appendix reports slowdowns against BF16 in some single-KV-head configurations. B200’s 4-bit path also uses certain floating-point representations whose matrix-operation behavior the hardware specification leaves undefined, although the authors’ checks passed. FOSCAR therefore offers evidence for a targeted optimization, not a universal speed or accuracy guarantee.