Routing AI queries could give datacenters room to cut power demand

A datacenter serving interactive language models faces a difficult demand-response problem: how can it reduce electricity use when the grid needs relief without postponing incoming requests? In the preprint Creating Power Flexibility in Datacenters via Query Routing, Shivam Patel and colleagues investigate whether routing requests between different language models can provide that flexibility.

Their central result comes from a trace-driven simulation, not a field deployment. For a modeled facility with 10,000 Nvidia H100 GPUs, they report an average power reduction of 1.19 megawatts, about 16% of mean baseline facility power, when all queries are assumed eligible for either model pool. In a simulated curtailment event, the response reached 90% of the desired reduction in approximately ten seconds.

A routing decision becomes a power control

The approach separates requests into three classes: those requiring the large model, those requiring the small model, and flexible requests that either can serve adequately. A controller changes the fraction of flexible requests assigned to the smaller model pool, subject to each pool’s available capacity.

That eligibility assumption is essential. The paper takes query–model suitability as an input from existing routing methods; it does not establish that every request can move to a smaller model without affecting answer quality. Its power-control analysis operates within the assignments assumed suitable.

Routing changes apply only to newly arriving requests. Responses already being generated continue on their original instances. The mechanism therefore avoids deliberately dropping or deferring requests, but power does not change instantaneously: outstanding work must finish before affected instances settle into their new operating state.

Predicting the available room

To decide which routing choices are feasible, the researchers build an analytical model connecting request arrival rates and prompt and response lengths to inference performance and power consumption.

They approximate average token-iteration time as a fixed overhead plus a term that grows with the number of concurrent requests. From this relationship, they estimate request residence time, throughput and serving capacity. Separately calibrated power profiles translate concurrency into GPU power demand.

The measurements show a sharp increase in power when an idle instance starts serving requests, followed by more gradual increases as concurrency grows. Consequently, consolidating work onto fewer active instances can consume less aggregate power than spreading it evenly. Instances without requests still draw idle power; they are not assumed to switch off.

The authors evaluate the framework across four GPU platforms and language models ranging from one billion to 70 billion parameters. They report approximately 2.8% mean absolute percentage error for estimated achievable power flexibility. This is not a universal error rate for every predicted quantity: the appendix reports substantially larger concurrency-estimation errors in some configurations, particularly near capacity.

What the datacenter simulation assumes

The large-scale study divides GPU resources between a large-model pool running Llama-3.1-70B across four GPUs per instance and a smaller-model pool running Mistral-24B on individual GPUs. It uses a week-long Azure conversation trace, scaling arrivals so the peak five-minute average reaches 60% of aggregate serving capacity.

Facility power includes an assumed additional 300 watts per GPU for non-GPU components and a power usage effectiveness multiplier of 1.2. The reported facility-level savings therefore depend on these overhead assumptions as well as the modeled inference workload.

With half the requests classified as flexible, average curtailment was about 0.6 megawatts. At least 0.32 megawatts was available for 95% of the simulated week. Making every request flexible increased the average reduction to 1.19 megawatts; that figure is conditional on the strongest eligibility assumption, not a guaranteed saving for arbitrary workloads.

Flexibility has boundaries

Available control depends on both eligible requests and spare capacity. At low arrival rates, there is less computational demand to reduce. At high rates, model pools approach saturation and have little room to accept rerouted work. The largest flexibility generally appears between these extremes.

The authors also simulate a four-hour curtailment event, showing that reduced power can be sustained under the examined workload. Their broader response-time measurements span approximately three to 18 seconds. They explicitly note that routing alone cannot meet sub-second response requirements, and that response speed does not establish qualification for a grid service.

The contribution is a way to estimate and select feasible power operating points for inference serving. Its practical promise rests on having requests that genuinely tolerate different model choices, sufficient serving headroom, and power profiles calibrated to the deployment—not on treating smaller models as interchangeable with larger ones for every task.