Keeping the right conversation in memory could make AI chatbots faster

Why remembering is not enough

A chatbot’s next request often includes the entire conversation so far. Before generating its answer, the language model processes that input in a stage called prefill. As the conversation grows, repeatedly processing its history becomes increasingly expensive.

Prefix caching saves intermediate computations, called key-value states, for previously processed tokens. When another request begins with exactly the same token sequence, the server can reuse those states rather than calculate them again. This is exact computational reuse, not retrieving an answer to a vaguely similar question. It does not change the model’s outputs.

The difficulty is deciding what to keep. GPU memory is limited, conversations have different lengths, and users may return immediately, much later, or never. A cache eviction policy decides which stored entries to remove when space runs out.

In the preprint Intent-Aware Caching for Efficient LLM Serving, Kazi Hasan Ibn Arif and colleagues propose making those decisions partly from the task a user is pursuing.

Predicting useful reuse

The researchers examined WildChat and ShareChat, datasets containing real chatbot conversations and timestamps. They found that conversation length and the delay between turns differed across task categories. Code generation and debugging tended to involve more turns than some other tasks. Code-generation conversations also showed longer inter-turn delays than casual chat. Delay patterns varied with turn number, too.

These are statistical patterns, not guarantees about an individual user. They nevertheless offer information that a conventional least-recently-used policy, or LRU, does not explicitly capture.

The proposed method, Intent-Aware Caching, or IAC, groups requests by predicted intent and turn number. For each group, it estimates the probability of another turn and models the distribution of waiting times until that turn arrives.

To build its intent classifier, the team used a large language model to help discover categories and label a separate collection of conversations. It then trained a smaller classifier using a frozen text-embedding model and a neural-network prediction head. The final taxonomy contains 30 categories. At serving time, classification uses up to the latest four user queries, rather than calling the large labeling model for every request.

A ranked decision, not a crystal ball

IAC ranks eviction candidates using three signals in a fixed order. First, it checks whether an entry’s age is below the estimated median inter-turn delay for its intent–turn group. Entries outside that window receive lower priority. Among entries tied on that test, it prefers those with a higher probability of continuation. Size breaks remaining ties, favoring larger entries.

The design takes inspiration from an offline reference policy that knows future reuse times. IAC has no such knowledge: it substitutes estimates from past conversations. The median delay worked better in the reported experiments than the mean or mode, balancing premature eviction against keeping stale entries too long.

What the experiments found

The researchers implemented IAC in the SGLang serving framework and tested language models ranging from 8 billion to 110 billion parameters, using configurations with up to four NVIDIA A100 GPUs.

On ShareChat, with cache capacity for 140,000 tokens, IAC achieved a 34.6% cache hit rate, compared with 12.2% for LRU and 18.8% for Learned Prefix Caching, a machine-learning baseline. Here, hit rate means the fraction of input tokens whose stored states were reused—not the fraction of requests answered entirely from cache.

On WildChat at the same capacity, IAC reached 24.8%, versus 13.8% for LRU and 17.1% for the learned baseline.

Better reuse also reduced waiting time. For Qwen-1.5-110B, mean time to the first output token fell from 6.98 seconds with LRU to 3.76 seconds with IAC. The paper reports reductions in this metric of up to 46.1% against LRU. That measures how soon an answer starts, not how long the complete answer takes to generate.

Where the evidence stops

These were controlled replay experiments, not a live deployment trial. For WildChat and ShareChat, the researchers preserved within-conversation delays but generated new-conversation arrival times to control load. Additional datasets lacked timestamps and therefore required synthetic timing.

The main evaluation used statistics estimated once from 40,000 earlier conversations. A separate rolling-window experiment refreshed statistics using only past conversations and retained an advantage over LRU, supporting—but not fully establishing—the method’s ability to handle changing workloads.

The classifier also has costs: the selected model achieved 75% accuracy on held-out model-labeled data and took about 20 milliseconds per unbatched classification. Finally, the study primarily concerns multi-turn chat. Its results do not establish equivalent gains for agents, tool-use pipelines, or other applications. The demonstrated contribution is better management of reusable computation, not smarter answers.