Reserving resources for AI agents when their workloads are unknown

The problem: a workload that unfolds as it runs

AI agents built around large language models can repeatedly choose tools, inspect results and update their working context. Their complete execution path is therefore unknown when a task arrives. A database lookup might resolve a request immediately, while another task could trigger further searches, code execution or retries.

In Uncertainty-Aware Resource Orchestration for Agentic AI Systems, Masoud Shokrnezhad and Tarik Taleb study the infrastructure problem this creates. Reserving too little computing or network capacity risks throttling and delays; reserving too much leaves capacity unused. Their arXiv preprint presents an algorithm called Conformal Reservation for Orchestration, or CRO. Its reported evaluation uses synthetic simulations, not a deployed agent platform.

The authors model each execution as a sequence of tool invocations, with compute demand, network demand and duration recorded at every step. These become pipeline-level totals. The objective is to minimize reservation cost while meeting separate probability targets for resource coverage and deadline satisfaction, subject to available capacity.

How past executions set the reservation

CRO maintains a sliding window of completed executions. Before dispatching another task, it uses those observations to calculate upper bounds for compute, bandwidth and latency.

Its central mechanism is a finite-sample correction to an empirical quantile. Rather than simply selecting the historical percentile matching the desired coverage, CRO selects a rank that accounts for the incoming task as an additional observation. For a resource-overflow tolerance of 10%, the intended coverage is at least 90%.

The paper expresses this calculation through deviations from rolling means. Its proof explains that subtracting and then adding the same mean leaves the resulting bound equal to an appropriately ranked historical observation. The method therefore does not require fitting a normal, Gamma or other specified distribution.

CRO also checks whether the latency bound fits the deadline and whether resource reservations fit node capacity. If either check fails, it returns an infeasible result rather than lowering the reservation to fit. Deferral, rerouting or rejection would then depend on operator policy. Too few observations for the requested probability level also produce an infeasible result.

What the simulations found

The experiments generate equally likely light, medium and heavy task types. Pipeline depths vary randomly, and the heavy category uses a negative-binomial depth model to represent greater variability. Per-step resource demands follow Gamma distributions; latency combines a compute-related term with exponential noise.

The authors compare CRO with four alternatives: an uncorrected empirical quantile, a normal approximation, a fitted Gamma distribution and a conservative one-sided statistical bound labeled Chebyshev. The reported setup uses 400 Monte Carlo trials and 1,000 test tasks per trial, with a default calibration window of 100 traces.

At a 90% compute-coverage target, CRO achieved reported coverage of 90.3%, compared with 89.4% for the uncorrected empirical quantile. The Chebyshev baseline reached 97.3%, reserving considerably more than the target required.

The corresponding normalized compute over-reservation was 1.81 for CRO and 3.58 for Chebyshev—approximately half as much excess reservation. This metric measures unused reserved compute relative to mean compute demand, not total infrastructure spending or energy consumption. Other methods recorded smaller excess reservations at that operating point but missed the coverage target.

With only ten calibration traces and the same 90% target, the authors report 91.1% coverage for CRO and 83.8% for the uncorrected quantile. All compared methods achieved perfect deadline-feasibility classification in the reported experiment, so that result does not distinguish CRO from the alternatives.

What the guarantees do—and do not—cover

The mathematical guarantee depends on exchangeability: the calibration traces and incoming execution must be statistically interchangeable. The authors treat approximate stationarity within the sliding window as the practical basis for this assumption. A sliding window alone does not establish protection against arbitrary workload shifts.

The guarantees are also marginal, covering compute, bandwidth and latency separately. They are not a promise that every requirement will hold simultaneously for every task, nor a demonstrated task-specific reliability guarantee after admission decisions.

The results support a narrower conclusion than universal scheduling superiority: CRO offered a favorable coverage–reservation tradeoff among the evaluated methods on the simulated workloads. The authors explicitly avoid claiming universal nonparametric optimality. Validation on real agent execution logs, production-platform integration, distribution-shift detection and joint reservation for concurrent tasks remain future work.