Grouping similar AI requests could reduce the hardware needed to serve them

Why mixing requests can be costly

Running a generative AI model after training is called inference. Providers want their graphics processing units, or GPUs, to produce useful responses while meeting users’ expectations for speed. The difficulty is that requests differ enormously in the resources they occupy and how long they run.

Long inputs and extended responses can consume substantial computing capacity and memory. When these requests share hardware with lighter ones, they can delay their neighbors. Adding GPUs may reduce those delays, but it can also leave expensive equipment underused.

Renée St. Amant, Anjaly Parayil, Ankur Mallick and Victor Rühle, all affiliated with Microsoft, investigate an alternative: workload isolation, meaning grouping requests with similar resource needs into separate serving pools. Here, isolation concerns workload allocation, not a demonstrated security boundary.

Their paper was submitted to arXiv on November 6, 2026. It is a preprint, and its findings should be treated as preliminary rather than established through peer review.

What the researchers measured

The production evaluation used one peak hour of traffic from a region, spanning many serving instances. Input processing and response generation ran together in a co-located configuration.

Requests were divided by input sequence length into three groups: roughly 70% short, 20% medium and 10% long. Sequence length counts tokens, the pieces of text processed by a language model.

The researchers examined batch sizes, GPU allocation and relative request costs. A batch allows a GPU to advance several responses together, sharing hardware costs across concurrent requests. Performance constraints covered both the wait for the first generated token and the time between subsequent tokens.

This is production-based evidence for a particular configuration, not a broad benchmark across many disclosed models and hardware platforms. The reported setup does not specify the model, GPU type, exact partition boundaries or total request count, limiting independent assessment and reproduction.

The reported gains

During the measured peak hour, the short-input pool achieved an average response-generation batch size 14 times larger than the long-input pool while remaining within interactive latency constraints.

Compared with serving without isolation, the authors report roughly a twofold reduction in GPU requirements for the workload, alongside an approximately halved mean GPU cost per request.

The allocation figures explain why separating requests mattered. Short inputs accounted for more than 70% of requests but used less than 20% of GPUs. Long inputs accounted for fewer than 9% of requests yet used more than 60% of GPUs. The resulting average GPU cost per long request was about 26 times that of a short request, before further pool-specific optimization.

These are workload-specific ratios, not universal prices for short and long prompts. Isolation does not inherently make an expensive request cheaper. Its central benefit is preventing costly requests from imposing the same operating constraints on the entire fleet.

Why the traffic mix matters

Separating costs also changes capacity planning. In a conditional calculation using the observed cost differences, the authors assume a fleet already at capacity and a 10% increase in total request volume.

If all additional requests were short, estimated fleet costs would rise by about 2.75%. If all were long, they would rise by about 71.52%. These are planning scenarios, not measured future growth outcomes. They illustrate why request counts alone may give a misleading picture of hardware needs.

The paper also frames unpredictable, path-dependent workloads as a problem of “non-ergodicity”: averages across requests may not reliably describe how resource use unfolds over time. That is the authors’ interpretation; a heavy-tailed cost distribution alone does not establish non-ergodicity.

Benefits still to be tested

More uniform pools could support clearer monitoring, tailored software and better matching between workloads and hardware generations. The paper discusses these as opportunities, rather than demonstrating every proposed benefit experimentally.

It uses output-token production and processed-request counts as proxies for utility and users affected. Those measures are informative operationally, but they do not directly measure answer quality, customer value or distinct users.

The practical limits

Too many narrowly defined pools can leave capacity idle while waiting for matching requests. Isolation also brings engineering, maintenance and caching tradeoffs.

Input length is known before execution, but output length is not. The evaluated workload had a strong relationship between the two; other workloads may not. A short prompt can trigger a long response, requiring prediction, rerouting or migration between pools.

The study therefore supports a specific conclusion: grouping similar requests can substantially improve hardware efficiency in the reported setting. Choosing reliable boundaries and maintaining those gains across changing traffic remain open challenges.