Why cancelling timed-out AI requests may not end a retry storm

A timeout tells a client to stop waiting. It does not necessarily tell the computer generating the answer to stop working. Retrying can therefore create a second execution of a request whose first execution is still running. Cancelling the abandoned execution seems like an obvious fix—but what happens if its replacement spends almost its entire deadline waiting in the same overloaded queue?

In a preprint submitted to arXiv on November 5, Xindan Zhang and Fei Wang investigate that problem using vLLM, software for serving large language models. Their experiments distinguish a call, the request an application wants answered, from an attempt, its initial submission or a retry, and an execution, the work created inside the model engine. That distinction matters: four attempts to obtain one answer can become four separate executions.

A short surge, a long disruption

The researchers tested Qwen2.5-7B-Instruct, Qwen3-14B and Llama-3.1-8B-Instruct, each running with vLLM 0.30.0 on one NVIDIA H100 GPU. Their synthetic workload used prompts averaging 12,000 tokens—the pieces of text a model processes—with a shared 2,000-token prefix. Clients stopped waiting after 120 seconds and could retry three times.

After calibrating each model's capacity, the team sent new calls at 60% of that capacity. A 107-second surge raised arrivals to 270%, followed by 720 seconds back at the lower rate. The question was whether service recovered after the surge, despite the backlog it left behind.

With retries enabled and cancellation disabled, none of the calls arriving during that post-surge period was delivered within the study's 840-second delivery horizon. All three models recovered during the observation period when retries were disabled. With retries and cancellation both enabled, cancellation improved delivery but did not restore service in any diagnostic run: roughly 13–38% of post-surge calls received complete responses.

This was not an inevitable result of any traffic increase. All three models recovered with retries enabled after a smaller surge to 150% of capacity. Results after an intermediate surge differed between models, and the researchers did not establish a precise failure threshold.

Cancellation arrives, but work still repeats

Detailed scheduler traces on Qwen2.5-7B-Instruct suggest how the failure persisted. Every recorded abort in the cancellation-enabled tracing runs followed the client's departure by less than half a second. Cancellation was reaching the engine promptly.

The difficulty was what came next. Retries that eventually ran typically began processing their prompts less than half a second before their own deadlines. The timing was consistent with a cycle in which attempts waited almost their entire lifetime, began work, then were aborted and replaced by another retry at the back of the queue.

Executions eventually aborted accounted for 79–85% of computed prompt tokens in these traces. At the larger processing-step size, about half of that aborted work belonged to steps already issued to the GPU when cancellation arrived; those steps still completed. These are prompt-token measurements, not direct measurements of GPU energy or kernel execution time.

The traces also exposed a monitoring limitation. vLLM's prompt-token counter records a prompt when its execution produces its first output token. Executions cancelled before that point can consume prompt processing without appearing in that counter. Across the traced runs, such executions accounted for about a third to half of computed prompt tokens.

Protecting new calls or finishing old ones

Two established controls restored service on all three models with cancellation disabled, but served different priorities. A retry budget restricted retries, shedding load while leaving many calls from the surge unanswered. Request coalescing instead attached a retry to its call's existing execution and replayed output already generated. It delivered every surge and post-surge call within 840 seconds, although some arrived after 120 seconds.

Immediate cancellation undermined coalescing in the Qwen2.5 tests: it destroyed the execution before the retry could reconnect. Waiting ten seconds before cancellation allowed retries to attach and restored service. However, no execution actually reached deferred cancellation in those runs, so cleanup after a client permanently disappears was not tested.

The findings remain specific to one engine version, one GPU type and synthetic, single-turn traffic with a fixed deadline and retry policy. They do not establish how other engines, multiple replicas or different clients would behave. Within this testbed, the practical distinction was clear: restoring timely service for new calls and eventually answering every call were different goals. Cancellation alone achieved neither while retries kept creating replacement work.