Why AI agents need a different approach to saving GPU power

AI assistants that use tools often work through a chain of steps: they receive a request, consult a tool, take in the result and continue reasoning. Each step can add to the context the language model must keep track of. That makes these “agentic” workloads different from a simple exchange of one prompt and one answer—and can complicate efforts to cut the power used by the GPUs serving them.

The paper “KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving,” by Yichao Yuan, Mosharaf Chowdhury and Nishil Talati, asks how a server can reduce power without letting response performance fall below its targets. The authors argue that power-management approaches designed around single-turn requests miss an important feature of agent serving: requests have histories that grow and change as the agent progresses.

Why turning down the GPU can backfire

A GPU’s operating frequency affects how quickly it can perform computations and how much power it uses. Lowering the frequency can save power, but it can also slow processing. In agent workloads, the model’s growing context occupies memory, and many active requests may compete for that space. The abstract reports that, under these conditions, lowering frequency can push a system into “thrashing”: memory pressure worsens, and performance and power efficiency both suffer.

The basic challenge is therefore not simply to run the GPU more slowly. A server needs to know whether it has enough memory headroom to do so safely, and whether requests should be handled together or sent to different instances.

A controller that tracks each agent

KAIROS treats agent context as a control signal. According to the abstract, it tracks requests at the agent level and adjusts three things together: GPU frequency, how many requests each instance handles concurrently, and how agents are placed across multiple instances. Its local controls respond to context growth and agent progress; its routing decisions aim to improve power efficiency while keeping memory stable.

In plain terms, the system tries to spend less power when memory has room, but avoid pushing an already pressured instance into a cycle of slowdown and worsening memory contention. Coordinating frequency, concurrency and placement is central to the proposal: changing one setting in isolation might not account for what is happening to an agent’s context elsewhere in the serving system.

What the authors report

The authors say they evaluated KAIROS on a range of software- and data-engineering agent tasks. They report an average power reduction of 27%, with reductions as high as 39.8%, while meeting the performance targets used in their evaluation.

Those figures describe the authors’ reported results, not a guarantee of the same savings in every deployment. The abstract does not specify the hardware, workload mix, baseline, exact performance targets or how the measurements were aggregated. Without those details, readers cannot tell how much the result depends on a particular setup or how it would transfer to different models, GPUs and real-world traffic.

What remains uncertain

This account is based on the paper’s abstract, not its full text. That means the methods and experimental details cannot be checked here, and the available material does not establish whether the work has undergone peer review. The reported power reductions should therefore be treated as claims from a preprint rather than settled evidence. The abstract also gives no comparison against named alternative systems and does not quantify trade-offs beyond saying that performance targets were met.

The paper’s central idea is nevertheless clear: power management for AI agents may need to respond to the state of an ongoing task, not just the current request. Whether KAIROS’s reported gains hold across broader hardware and production workloads is a question the abstract alone cannot answer.