Turning AI agent traces into reusable workflows

The idea: compile the decision-making

A language-model agent can read a standard operating procedure (SOP), use tools and choose what to do next. But when the process repeats, the agent may need another model call at every step. ReTrace, described in an arXiv preprint marked “under review,” aims to shift that planning effort to setup. It builds a workflow in which code selects the next action, rather than asking an AI model to plan each run.

A log of tool calls does not necessarily explain why an agent chose them. ReTrace therefore uses both the calls and the explanations the agent provides alongside them. The authors say the explanations can help identify the conditions behind an action and alternatives the agent considered but did not take. Tool records provide the call’s inputs and results; the system uses them to identify input values and where those inputs came from.

From examples to workflow

ReTrace has three stages. First, a language model examines each logged step and proposes a condition, action, tool and input sources. A customer ID, for example, can be treated as a variable supplied by each new task rather than copied from an example.

Next, the system compares summaries of the runs with the written procedure and merges them into a compact rule document. It records evidence for its rules and marks some branches “unwitnessed” when no example followed them. Finally, a model converts the document into a workflow for the target platform. At run time, code checks which condition applies and selects the next step. Individual tools may still call models—for instance, to classify a document—but the workflow does not ask a model to choose what step comes next.

The researchers tested ReTrace on 79 procedures from a private business-automation platform and two public benchmarks. Public tests used the telecom portion of τ²-Bench and eight of SOP-Bench’s 14 domains. Workflows were built using training examples and assessed on separate inputs. Some were also revised using feedback about failures on training inputs.

Outcomes depend on the measure

On the production-platform procedures, ReTrace workflows had 93.8% completion, compared with 96.8% for the original agent, and cost 24% less per task on average. Time results were mixed: median time fell from 68 to 37 seconds, but mean time rose from 153 to 163 seconds. The authors attribute the mean increase partly to web-automation calls: successful calls can take a long time, while failed calls may finish quickly. The platform’s completion flag also sometimes counted runs as complete despite unsuccessful tool outcomes, so completion alone is not a dependable measure of task success.

On τ²-Bench, ReTrace reached 75.8% task success, compared with 80.6% for the agent. Average execution time was 1.3 seconds rather than 170 seconds. After three rounds of repair using training failures, a version built from two traces reached 81.2%. That is close to the agent’s score, but the confidence interval was too wide to establish that the two performed equally.

On SOP-Bench, unrepaired ReTrace scored 80.6%, below few-shot workflow generation at 83.6% and the agent at 85.5%. Repair using labeled training failures raised ReTrace to 87.8%. This is not a like-for-like comparison with unrepaired methods, because repair received feedback about training labels.

The agent’s explanations helped in some settings but not all. Removing them lowered scores on τ²-Bench and the production procedures; on SOP-Bench, the version without explanations scored slightly higher. The authors suggest that for short, lookup-like procedures, tool calls may already provide much of the useful information.

What remains uncertain

These findings suggest a trade-off, not a general replacement for agents. Compiled workflows can greatly reduce execution time on the tested telecom benchmark, but results depend on the procedure, examples and tools. The explanations ReTrace relies on may be mistaken or may not reflect the real reason for an action. Changes to an SOP or its tools may require rebuilding the workflow, and evaluation did not test transfer to entirely new procedures.

The validation checks also differ by dataset. For public benchmarks, generated candidates were checked for planning calls and re-executed on training inputs; the authors say this does not establish that the recovered rules are correct. Enterprise workflows were not re-executed in the same sandbox: the platform generated a typed workflow graph, and the stated guarantee was that it contained no planner node. Enterprise completion itself can misrepresent whether the task goal was achieved.

Other limits include restricted benchmark coverage and sensitivity to trace selection; results for an individual SOP-Bench procedure varied substantially across trace samples. The private production data cannot be independently inspected. The preprint’s findings should therefore be treated as provisional. ReTrace is a method for turning repeated agent decisions into reusable programs, not proof that those decisions are sound. The authors recommend reviewing recovered rules, checking tool outcomes, limiting tool access and retaining human review for consequential actions.