Verified tool-health snapshots improve AI agents’ first choices in controlled tests

The gap between capability and current conditions

An AI agent may know that two tools can retrieve the same record without knowing that one is currently slow or unavailable. Manu Agrawal and Shrey Nagpal call this gap operational blindness: the execution software has useful monitoring information, but the model choosing an action cannot see it.

In their preprint, Agent Harness Awareness: Turning Verified Telemetry into Operational Control for Tool-Using Agents, the independent researchers propose giving models a compact, current snapshot of relevant operational conditions. Their experiments test a narrow question: when tools offer equivalent functionality, does verified health information improve the model’s first choice?

The reported results support that mechanism in controlled tools-and-network settings. They do not establish general production reliability, and the supplied source identifies the work as a preprint rather than a peer-reviewed publication.

Information without transferring authority

The proposed architecture separates model choice from infrastructure authority. The host—the software that runs the agent and its tools—measures recent outcomes. A deterministic reducer selects relevant facts, limits and redacts their contents, and attaches provenance and freshness information.

The model uses this snapshot alongside the task and tool descriptions. The host remains responsible for rechecking current conditions, enforcing constraints, and executing or safely rejecting the proposed action. The model does not gain authority to alter telemetry or bypass enforcement.

The authors call the interface the Substrate State Telemetry contract. Their tested snapshots include recent successes, latency estimates, consecutive failures, error information, circuit state, and observation age. Runtime constraints, hardware pressure, and economics belong to the broader proposal but were not evaluated here.

A controlled comparison of three information conditions

The first study used ToolRoute, a benchmark in which models retrieved a read-only customer record through either of two functionally equivalent routes, or chose to wait. It comprised 216 independent decisions across GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, six seeds, four predeclared environment states, and three information conditions.

Condition A supplied only the task and tool descriptions. Condition B added a telemetry-shaped envelope with equal neutral values for both routes. Condition C used the same structure but supplied fresh, host-owned health information. Comparing B with C helps separate the effect of useful state values from simply adding a structured interface. However, the control did not exactly match provider token counts.

Each decision began from an independent checkpoint, with no request retries. This design isolates initial selection rather than recovery after failure.

What the reported improvements mean

Against the neutral control, verified state increased first-dispatch completion from 50 of 72 decisions, or 69.4%, to 69 of 72, or 95.8%. Mean “observable policy regret” fell from 4,251.54 to 131.12 milliseconds, a reported 96.9% reduction.

Regret is not measured request duration. It represents excess operational cost relative to the best route identifiable from the disclosed monitoring facts. The scoring formula combines latency estimates with fixed failure and open-circuit penalties. Consequently, a large reduction in regret should not be read as an equivalent reduction in real-world response time.

All 69 verified-state decisions that selected an active route chose an observable-best route. The other three chose to wait when both routes were degraded. Waiting counted as failure to complete the retrieval task and accounted for all remaining regret in that condition.

A separate live-path replication

The second study tested 27 independent decisions using live local TCP/HTTP execution. OpenTelemetry client spans supplied monitoring observations, while Toxiproxy introduced latency, connection failures, or HTTP 503 errors. The chosen route was then executed through the same live proxy.

Verified state produced successful actions in 9 of 9 cases, compared with 5 of 9 for the neutral control. Mean regret fell from 8,956.37 milliseconds to zero. Zero regret means no excess observable operational cost—not an instantaneous HTTP request.

The authors keep this study separate from the larger experiment. Nine verified-state cases demonstrate the mechanism across the tested model-and-fault combinations, not a population-wide guarantee; the reported 95% completion interval spans 70.1% to 100%.

Where the evidence stops

The routes were deliberately equivalent, the task was read-only, and retries were excluded. The studies therefore do not establish benefits for heterogeneous providers, non-idempotent actions, prolonged workflows, or retry-aware recovery. Snapshot decay and freshness-aware dispatch also remain open tests.

The authors explicitly argue that routine failover should stay in deterministic middleware. Their broader hypothesis is that verified state could help agents make task-dependent choices among providers, sources, or plans. These experiments establish a narrower starting point: selected host telemetry can materially improve first-dispatch choice when capability is held constant.