Old AI Memory, New Model: Useful but Not the Same

A KV cache is a model’s working record of text it has already processed. If a prompt contains a long conversation or a set of documents, reusing that record can avoid doing the same preliminary computation again. The complication is that a model update changes how text is represented internally. The new model may be able to use its predecessor’s cached state, but that does not mean it will respond exactly as it would with a cache it made itself.

Zhikun Tao, Fangyong Jin and Junping Zhao examine this question in a technical report posted as an arXiv preprint. They study direct cache reuse across four model updates, focusing their broader question on whether an updated model can continue from a predecessor’s cached prompt without translating the cache or recomputing selected parts. Their detailed analysis centers on DeepSeek-V4-Flash and Flash-0731.

The researchers compare three setups: an older model using its own cache, an updated model using its own cache, and the updated model continuing from the older model’s cache. To test retrieval, they use “needle in a haystack” prompts: long passages containing many key–value records, followed by a question about one record. They also test natural-language question-answering benchmarks, measure time to first token, compare models’ next-token probability distributions, and inspect selected internal computations. In the main question-answering speed tests, the source cache was prepared in advance.

The retrieval results vary markedly by model pair. For DeepSeek-V4.1-Flash updated to Flash-1017, reused-cache accuracy stayed close to the updated model’s own-cache result through the tested 32,000-token prompts. In contrast, for DeepSeek-V4-Flash to Flash-0731, retrieval became much harder as prompts grew: at 32,000 tokens, the updated model scored 25.52% with its own cache and 31.25% with the older cache. Reuse helped on average in that comparison, but remained far below the older model’s 55.73%. For Qwen3-30B-A3B to Instruct-2507, reuse reduced retrieval accuracy at every tested length. These results rule out a simple guarantee that caches from successive versions will work equally well.

In the DeepSeek question-answering tests, prepared-cache reuse shortened median time to first token on three benchmarks with longer prompts: by factors of 5.08 on HotpotQA, 2.10 on 2WikiMQA, and 2.51 on MultiFieldQA-en. Scores did not move uniformly. MultiFieldQA-en’s F1 score was nearly unchanged, while HotpotQA and 2WikiMQA scores fell by about 3 and 7 points, respectively. On BABILong, whose prompts were shorter, reuse was slower overall: the reported speedup was 0.86×. The timing comparison excludes source-model prefill and model startup, so it describes reuse when the cache is already available—not the full cost of creating and managing it.

Speed and answer quality are not the only differences. The researchers found cases where the reused-cache and native runs began with the same answer token but later diverged, either adding extra wording or giving different content. Their measure of Functional Cache Divergence compares the complete next-token probability distributions under the two cache conditions. For the Flash update, divergence grew with prompt length on the synthetic retrieval task, while remaining lower and relatively steady during natural-text continuation. A large distribution shift did not always mean a worse answer: it accompanied both corrected and lost retrieval answers in the tested examples.

The authors also report evidence that cached states can convey more than facts. In a separate experiment, Flash-generated caches helped both an earlier Base model and Flash-0731 retrieve information. The prompt included instructions about how to answer, so the cache reflected the model’s processing of both records and response instructions. This suggests one reason an updated model might answer differently even when the underlying text is unchanged.

The takeaway is not that old caches are universally safe or unsafe. They can preserve useful task information and save computation, but they need not reproduce the updated model’s behavior. The evidence is limited to specific model pairs, benchmark tasks, prompt formats, and serving setups; several analyses use selected diagnostic examples. The report is a preprint, not evidence of peer-reviewed consensus. Its authors also call for testing more tasks and decoding settings, and the reported speedups do not include every cost a deployed system might incur.