When a brief traffic spike leaves an AI server struggling

An AI coding agent might ask a model to inspect a file, run a tool, read the result and ask the model again. Each turn carries a growing conversation history. A server can keep the history’s attention data—called key-value, or KV, states—in a prefix cache. When the next turn shares that prefix, the server can process mostly the new material. If the needed states are no longer available, it must recompute much more of the context.

Wen Huang and Jack Liu’s arXiv preprint, Bursts That Outlast Themselves, asks whether a short rise in incoming agent sessions can have effects that persist after arrivals return to their usual level. The authors report that it sometimes can. Their proposed explanation is not simply that a burst knocks the cache out of place: slower service keeps more sessions in the system, whose combined KV demand can exceed the pool. The authors infer that prefixes are then evicted before later turns need them, leading to more recomputation and slower service. They did not directly observe individual prefix-eviction events; the engine does not report them.

Testing the feedback loop

The researchers paired runs on vLLM, using Qwen3-8B on a single AMD MI300X GPU. Each pair received the same base sessions at the same times; one run also received extra arrivals for five minutes. In 9 of 12 pairs, degradation persisted only in the burst run. Two pairs degraded in both runs, and one in neither. The authors also report that two no-burst runs degraded spontaneously, underscoring that a burst raised the likelihood of entering the degraded regime but did not determine the outcome.

In degraded 8B runs, the prefix hit ratio fell to around 10%, and the growing backlog could exceed 100 sessions. Throughput in this regime was about 0.4 turns per second, similar to the engine’s capacity with caching disabled at comparable load. The authors’ measurements support a cycle: more concurrent sessions increase the working set competing for the KV pool; cache misses require recomputing long contexts; and that extra work further slows service. The specific link from misses to individual prefixes being evicted is an inference, rather than a directly measured event.

Controls help interpret the result. Without prefix caching, the tested absolute bursts produced backlogs that later drained, although some were still draining at the end of the run. In tests with smaller, rate-proportional bursts, paired runs ended with small backlogs. The researchers also tested SGLang: it recovered from the specific 2×, two-minute burst tested, which collapsed vLLM in some runs, but a larger tested trigger produced persistent degradation. This shows that the threshold can depend on the engine; it does not establish how all engines respond.

The team also replayed the lengths and timing of SWE-bench agent traces with Qwen3-32B. At one tested arrival rate, one of four burst runs persistently degraded and another was partly degraded; a longer no-burst run later degraded on its own. This is limited evidence for history dependence in that configuration. The replay preserved workload shapes, not prompt content, and did not execute agents.

Recovery and prevention

After collapse, simply limiting new admissions was not enough in most matched 8B runs: among seven runs that had all collapsed at the start of control, only one recovered with admission-only throttling. A session-cap controller that also parked admitted sessions at turn boundaries recovered all seven within six minutes. The capacity-based budgeter recovered all seven in under five minutes. Parking temporarily pauses some sessions so the remaining active sessions can fit the KV pool and regain cache hits; it does not erase the queued work.

The researchers also report that the budgeter prevented collapse in the trigger conditions and configurations they tested. In a 32B trace experiment, it restored high cache-hit rates, but the queue still grew because arrivals slightly exceeded the server’s completion rate. Preventing cache collapse cannot create additional serving capacity.

What remains uncertain

These experiments used AMD MI300X GPUs, one replica, and particular versions of vLLM and SGLang with the tested Qwen models. The study used synthetic sessions as well as trace-shaped replays, and finite experiment windows. It did not test production deployment, routing, autoscaling or client retries. The authors also note that real production contexts can be longer, but did not measure how that affects this failure. Results therefore do not establish how often the problem occurs across services, hardware or workloads.

The paper is an arXiv preprint, not a peer-reviewed publication. Its findings suggest that, for workloads relying on cached agent history, monitoring cache hits and KV demand may help operators spot trouble. They also indicate that managing sessions already in progress can matter, alongside controlling new arrivals. Whether these approaches generalize remains to be tested.