AMD’s InferaSim aims to cut the GPU cost of tuning LLM serving

Running a large language model efficiently involves more than choosing a fast GPU. Operators must decide how to divide the model across processors, batch requests, manage memory and balance response speed against total output. Testing those combinations consumes the same hardware needed to serve users.

In the November 2, 2026 preprint ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration (arXiv:2611.04032v1), Anshu Raina and colleagues at AMD propose replacing many hardware trials with predictions. Their results suggest a way to narrow deployment searches—not a guarantee that a predicted winner will satisfy production requirements.

Measure a representative run

The framework starts with an “anchor”: a short measurement of complete processing steps in a serving engine. Rather than timing every low-level GPU operation separately, it captures their combined cost.

The underlying distinction is between changes that alter the operations being executed and changes that mostly redistribute their workload. Batch size and some parallel layouts can be projected from an existing anchor. A different engine, GPU type, numerical precision or attention implementation generally requires a matching measurement.

A crucial modeling choice is to preserve fixed overhead. Doubling the GPUs does not halve the time spent launching operations, synchronizing or scheduling. In one DeepSeek-V4-Pro transfer from four to eight GPUs, the authors report 0.9% median decode-step prediction error with their method, compared with 44.1% when simply halving the measured time. That is a specific component check, not the framework’s overall accuracy.

For larger deployments, InferaSim reduces the target to a configuration measurable on one node, then analytically restores pipeline stages, communication and other removed components.

Simulate the requests, too

Hardware speed alone cannot explain the experience of users sharing a server. Processing a new prompt can interrupt ongoing output generation, while coding agents pause between turns and reuse earlier conversation prefixes.

InferaSim therefore simulates individual requests, queues, batching and cached intermediate information known as the key–value cache. It also models moving cached information into host memory and separating prompt processing, or prefill, from output generation, or decode.

A tuning agent searches these predictions. Its LLM component proposes settings; it does not invent performance estimates. Importantly, the reported large search experiment exercised only the deterministic seed stage, without LLM calls. Those results do not establish an additional benefit from the LLM-guided search loop.

What the measurements show

All hardware measurements used AMD Instinct MI355X GPUs, with vLLM, SGLang and AMD’s ATOM serving engines.

For one fixed-length benchmark collection, 11 anchors required 9.1 GPU-hours, compared with 204 GPU-hours to measure the 226 configurations they covered. GPU-hours count aggregate device use, not elapsed wall-clock time. The paper uses 75 cached anchors overall; 11 is not the total for every experiment.

Accuracy was scored separately on 172 fixed-length configurations passing the measurement filters. Median absolute percentage errors were 8.1% for time per output token, 8.8% for throughput and 13.6% for time to first token. These medians do not bound individual errors.

On 95 covered, held-out configurations replaying the AgentX coding trace, throughput had an error factor of 1.08, versus 1.14 for inter-token latency and 1.23 for time to first token. This geometric-mean factor measures multiplicative misses in either direction; it does not mean every throughput prediction was within 8%.

For 159 fixed-length deployments with separate prefill and decode pools, median throughput-per-GPU error was 5.5%, despite none being an anchor. Median first-token latency error was roughly 17%, again showing that predicting response startup was harder than predicting output capacity.

The authors also estimate that directly measuring the candidates in 192 searches would require about 25,000 GPU-hours. The prediction searches took 3–8 CPU-hours. The GPU figure is an extrapolated trial cost, not a completed exhaustive benchmark.

A shortlist, not an automatic deployment guarantee

The deployment planner’s five-candidate shortlist contained the measured best configuration in 21 of 23 feasible model–latency-tier slots, and one within 12% of the best throughput in 22. Every feasible slot had a shortlisted candidate meeting its target on hardware. However, the planner’s single selected configuration missed the latency tier in two slots.

Several boundaries matter. Anchor compatibility is checked through recorded settings, not verified by tracing the target’s actual GPU operations. Predictions outside measured ranges are flagged. Measurement-free mode is intended for ranking within an engine, rather than establishing absolute performance across engines.

The evaluation does not establish accuracy on NVIDIA hardware; the authors leave that testing to future work. They also state that anchors, measured datasets and reproduction scripts will be released with the camera-ready version.

The practical proposition is therefore narrower than the title might imply: spend less GPU time identifying promising candidates, then confirm the shortlist on hardware before deployment.