Giving AI agents a shared budget for risky actions

A tool-using AI agent does more than produce an answer. Its workflow can include searches, memory retrieval, planning and actions in an external environment. That creates several opportunities for malicious instructions to influence what happens next.

In a November 2026 arXiv preprint, Shilong Pan and colleagues propose Tail-risk-aware Agent Safety, or TAS. Their central idea is to track risk across dependent steps while paying particular attention to the high-cost end of the estimated outcome distribution. The results are benchmark evidence, not proof that an agent is safe in deployment. The supplied source does not establish peer-reviewed status.

Why averages can miss danger

The researchers distinguish average risk from tail risk: outcomes that are uncommon but potentially costly. If most executions receive very low safety-cost scores, their average can remain modest even when a small fraction receive extremely high scores.

TAS therefore asks a different question from whether a proposed step looks acceptable on average. It examines the costly end of a sampled distribution and gives greater weight to its highest-scoring outcomes.

Those costs are constructed risk scores, not measured financial losses or direct estimates of real-world harm. That distinction matters when interpreting both the method and its results.

Tracking the workflow

TAS represents each step with a safety state containing the original query, accumulated workflow observations and accumulated safety cost. The researchers describe this as a Markov decision process: the next state depends on the current state and the chosen action.

This representation makes earlier activity available to later safety decisions. The framework also maintains one shared budget. As assessed costs accumulate, less budget remains for subsequent steps, rather than every step receiving an independent allowance.

To assess a candidate action, TAS samples possible outputs and scores them. Its evaluator combines rules about tool names, arguments, sensitive information and external destinations with a language-model-based semantic assessment. The reported configuration uses GPT-5-mini for that assessment.

The sampled costs are sorted, and a spectral risk measure takes a weighted average of their upper tail. In the main configuration, TAS draws 200 candidate outputs and uses a tail cutoff of 0.95, concentrating attention on approximately the highest-cost five percent of samples.

If the estimated tail score is below the remaining budget, the step can proceed. If it exactly reaches the budget, the step is allowed but the workflow stops afterward. If it exceeds the budget, TAS rejects the proposed step and tries an alternative action. After unsuccessful retries, it terminates with a predefined safe fallback response.

What the experiments found

The study compares TAS with eleven baseline defenses across Agent Security Bench, Agent-SafetyBench and AgentDojo, with different comparisons suited to different attack settings.

On Agent Security Bench, the main table reports average direct prompt-injection attack success falling from 76.89 percent without a defense to 45.15 percent with TAS across four model backbones. For indirect prompt injection, the corresponding figures are 48.60 and 29.73 percent. Performance without attacks averages 61.81 percent without defense and 63.63 percent with TAS.

On AgentDojo, GPT-4o attack success falls from 47.51 to 3.93 percent, while reported utility rises from 56.74 to 63.32 percent. However, the open-model results illustrate why resistance alone is insufficient: their attack success is already near zero without TAS, but utility is only around nine to eleven percent.

Removing components generally worsens results. Replacing tail-sensitive scoring with an overall average, removing the shared constraint or using just one sample all reduces protection in the reported tests. These comparisons support the design within the tested settings.

The price and limits of protection

TAS requires more computation. The paper reports overall computational overhead of 2.34 times the undefended workflow, mostly attributable to sampling. Larger sample counts further reduce attack success in one experiment, but increase cost sharply.

Protection also depends on what the evaluator recognizes. More samples cannot by themselves establish that its risk scores capture every dangerous outcome. The authors acknowledge possible evaluator bias, unnecessary refusals and missed risks. Their tail-risk visualizations use the same scorer across methods, so they demonstrate lower assessed costs rather than independently measured real-world damage.

Finally, the implemented decisions check sampled next-step costs against a remaining budget. These experiments do not establish a universal bound on catastrophic workflow failures. TAS offers evidence for a practical design principle: assess actions in context, share constraints across steps and look beyond average outcomes—while keeping the evaluator’s uncertainty visible.