VerKV checks compressed AI memory before fetching more

Why long conversations become a memory problem

When a language model generates an answer, it repeatedly consults numerical representations of earlier tokens, the pieces into which text is divided. These stored representations, called keys and values, form the KV cache. Attention uses them to calculate which earlier information contributes to the next step.

The cache grows with context length. Keeping everything in fast GPU memory can become impractical; moving it to CPU memory preserves the information but introduces transfers that can slow generation. Keeping only selected entries on the GPU avoids those transfers, but a selection made before future queries are known can discard something important.

In the arXiv preprint Verify Before You Fetch, Devanshu Jha and colleagues propose VerKV: retain a small GPU-resident sample, keep the full cache accessible in CPU memory, and estimate whether additional entries are needed for each attention computation. This briefing draws on the supplied full-paper extraction; the work combines mathematical analysis with computational experiments, not an independently validated production deployment.

Sample first, retrieve selectively

VerKV begins with PoissonComp, which converts existing token-importance scores into probabilities of retaining cache entries. Highly ranked entries can be retained with certainty; others are sampled independently. A small set of initial and recent tokens also stays available.

Unlike simply keeping the highest-scoring entries, this randomized selection supports statistical estimates of what the sample misses. Sampled contributions receive weights related to their inclusion probabilities. The method can also reduce those weights to trade some systematic bias for lower sampling variability.

The GPU additionally holds a compact sketch of the context’s keys. For a new query, KVFetch uses this sketch and the sampling probabilities to estimate approximation error. If its calculated bound exceeds the chosen tolerance, it selects additional contributions to evaluate exactly, fetching their cache entries when they are not already resident. Its practical selection procedure is greedy, rather than a demonstrated globally optimal retrieval strategy.

The intended benefit is selective data movement—not deletion of the original context. The full cache remains available in host memory.

What the experiments found

The researchers evaluated Qwen3-4B, Qwen3.5-9B and Ministral-3-8B on RULER and LOFT tasks at approximately 32,000 and 128,000 tokens. These tests include finding information buried in context and answering questions using supplied material. Each RULER task used 50 examples; each LOFT task used 25 questions.

At matched GPU cache budgets, VerKV generally preserved much more benchmark quality than KVzip and Fast KVzip, two methods that retain entries without subsequent query-specific fetching. Across both context lengths, the paper’s aggregate table reports roughly 91–100% of full-cache performance at eightfold compression, depending on the model. At 32-fold compression, the range falls to roughly 80–95%.

These percentages are benchmark scores relative to the full-cache baseline, not percentages of answers guaranteed correct. Likewise, the eightfold to 32-fold memory reductions concern GPU-resident KV state, not total model memory. The accounting includes the sketch, while the full cache still requires host storage.

In a separate vLLM implementation on one NVIDIA H100 GPU, Qwen3-4B generated tokens about 1.6 times as fast as a baseline that offloaded the full cache. On RULER-32K, quality scored 86.7 against 88.0 for that baseline. This is a comparison with full-cache offloading, not evidence that VerKV outperforms keeping the entire cache on the GPU.

The guarantee has an important boundary

The paper’s formal analysis focuses on the denominator of attention: the sum used to normalize attention weights. Its probability bound assumes exact sketch weights and a fixed query independent of the initial random sample. It is not an end-to-end guarantee about answer accuracy or an entire generated conversation.

The deployed sketch instead uses heavily compressed, two-bit key information. Its errors are not included in the theoretical certificate. In the reported Qwen3-4B error-adherence experiment, a 5% error tolerance and 10% permitted failure probability produced violations in about 18% of decisions with the approximate sketch. An idealized perfect sketch kept violations below 1%.

Verification also costs computation. The current selection kernel limits speed, and batching provides relatively modest gains. Longer contexts and some harder tasks retain larger quality gaps with the approximate sketch.

VerKV therefore demonstrates a useful hybrid strategy: sampled memory plus adaptive retrieval can substantially outperform static eviction. Its strongest unresolved issue is making the practical error estimate as dependable as the idealized mathematical bound.