Signals: Diagnosing AI Agents Beyond Successful Tool Calls

When working software produces the wrong outcome

A tool call can succeed while an AI agent fails its task. The agent might choose the wrong action, ignore useful information, repeat unnecessary work, or announce success despite evidence that the task remains unfinished.

In Signals: Continuous Diagnostics of AI Agents, Amey Rambatla and five Cisco colleagues propose a method for diagnosing these problems from execution records. Their starting point is that records of model interactions, retrieved information, tool calls, and results need interpretation against the user’s request and applicable requirements. Successful service responses alone cannot establish correct behavior.

Turning observations into engineering issues

Signals distinguishes three units: a session is one recorded execution; an occurrence is an individual problem instance; and a Signal groups related occurrences into an issue a developer can investigate and address.

That grouping matters. Repeated instances of the same argument-construction problem may need one fix. Conversely, two problems with the same diagnostic label may require separate interventions. A shared label does not, by itself, establish a shared cause.

The method uses a taxonomy of 45 problem types across eight families. These cover instructions, knowledge and retrieval, reasoning and output, tool integration, workflow coordination, runtime reliability, safety, and resource efficiency.

The distinctions are practical. Sending invalid arguments, receiving an invalid tool result, and ignoring a valid result can all disrupt a task, but require different evidence and corrective actions. The authors require both the governing requirement and evidence of the behavior that violated it.

Failure, exposure, or waste?

Signals separates observed failures from risk exposures and inefficiencies. A realized defect has an observed adverse consequence. A risk exposure requires evidence of a reachable harmful path, even when harm was not observed. An inefficiency leaves the outcome acceptable but adds avoidable work without a supported benefit.

These categories prevent shortcuts in diagnosis. Suspicious-looking input alone does not prove a security failure. Missing records do not prove that an action never happened. Repeated calls may be necessary for verification, recovery, or pagination rather than evidence of waste.

Severity follows the same separation. The method scores observed consequences, reachable potential consequences, or avoidable waste, then adds points for the number and share of affected sessions. A model interprets the evidence; code applies fixed values and arithmetic. When a Signal contains multiple categories, its overall severity uses the highest category score rather than adding or averaging them.

A false conflict in insurance intake

An audited synthetic example shows why the distinctions matter. An agent compared a submission containing POL123 with an extracted value, Policy POL123. Although the second string merely added a field label to the same identifier, the workflow reported a conflict, sent the request for human review, and did not register the case.

The authors diagnosed a factual or logical error. The routing and missing registration demonstrated its effect; they did not independently establish additional defects or financial harm.

Their proposed remedy was to compare normalized identifiers while preserving genuine differences. Suggested checks included equivalent forms, different identifiers, and missing values. The audit did not include an implemented repair, and human review remained appropriate for genuine conflicts.

What the benchmark results show

The researchers evaluated Signals on recorded public benchmark executions spanning coding, tool use, coordination, and security. LLM judges created and reviewed reference labels under human oversight. The detector received execution evidence but not benchmark outcomes or reference labels, and used GPT-6.1 Sol.

For matching predicted Signals to reference issues, the pipeline achieved a reported 91.43% F1, compared with 73.69% for a one-pass control using the same model and complete taxonomy. That is a 17.75-percentage-point improvement. Precision was 92.90%, and recall was 90.01%.

Those scores have a specific meaning. Matching checked agreement on components and problem-type tags, not diagnosis wording, supporting evidence, severity, or repair correctness. Occurrence matching could accept a shared type among three ranked candidates, and the rules could connect different events involving the same component.

Diagnosis is not proof of repair

In separate robustness runs, agreement on whether a session contained a finding was 97.0–98.5%; removing some telemetry reduced it to 74.2%. These were session-level comparisons, not measures of agreement on every individual problem.

The authors also caution that selected benchmarks, correlated tasks, and exposure to benchmark families during taxonomy development constrain generalization. They did not establish production cost savings or reduced diagnosis time.

Signals therefore offers a structured route from execution evidence to proposed engineering action—not demonstrated automatic repair. The authors stress that one corrected run is insufficient: developers must check affected cases and previously successful behavior before concluding that a change worked.