Cached Agent Traces Help Predict Benchmark Scores From Fewer Tasks

Evaluating an AI agent involves more than asking questions and checking answers. Software and command-line tasks require execution environments, tool use and potentially lengthy interactions. In the November 4, 2026 preprint Task-Efficient Agent Evaluation Using Cached Traces, Ben Johnson and Hayden Helm investigate how previously collected execution logs can make these evaluations more task-efficient.

Their goal is to estimate a new system’s full benchmark score from a small number of task executions. The approach depends on a collection of reference systems whose traces and benchmark scores are already available. It does not eliminate the need to build that reference collection.

Turning logs into comparable behavior

A trace records an agent’s activity during a task: plans, tool calls, file changes, intermediate results and errors. Directly converting that log into a numerical representation can capture distinctive formatting or tooling rather than the behavior most useful for predicting performance.

The authors propose trubric, short for “trace rubric.” First, a language model uses a task description to create task-specific criteria under shared headings. The main experiments use six: understanding, localization, reproduction, editing, verification and final state.

A language model then summarizes each trace against those criteria. A text encoder converts each field’s summary into numbers, and the field representations are joined together. This makes different systems’ executions comparable along the same behavioral dimensions.

An example concerns a Django username-validation bug. The summaries distinguish an agent that checked its patch with a custom script from another that also reproduced the original problem and ran the existing test suite. Those differences provide information beyond whether the benchmark ultimately marked the execution correct.

Across seven text encoders, the authors’ classification experiments found that task-specific summaries retained task and correctness information while reducing recoverable information about system identity, model family and harness—the software framework running the agent. They did not eliminate those identity signals entirely.

Predicting scores from a behavioral map

The next step places systems in a low-dimensional numerical space based on similarities between their represented traces. A regression model learns how reference systems’ positions relate to their benchmark scores, then predicts the target system’s score from its position.

This trace-based predictor does not require correctness labels for the target’s executed tasks, although it still needs scored reference systems. The authors also combine it with item response theory, or IRT, which estimates ability from task outcomes while accounting for task difficulty and how informative each task is. That combined estimator uses both traces and correctness outcomes.

The study analyzes 107 SWE-bench Verified systems using a 100-task panel, while predicting official scores across all 500 benchmark tasks. For Terminal-Bench 2.0, it analyzes 65 systems with at least 90% public trace coverage across 89 tasks. It uses one trial’s trace and outcome per system-task pair to predict official scores averaged over five trials.

Evaluation protocols exclude either the target system alone, systems sharing its model family, or systems sharing its harness. These exclusions test prediction when closely related references are unavailable.

Where the gains appeared

Under the model-family exclusion protocol, with one randomly selected task, combining IRT and trubric reduced mean absolute error from 0.127 to 0.081 on SWE-bench Verified and from 0.137 to 0.122 on Terminal-Bench. On a percentage-score scale, those errors correspond to 12.7 versus 8.1 percentage points and 13.7 versus 12.2 points, respectively.

With five random tasks, the combined method achieved errors of 0.063 and 0.090 on the two benchmarks. Selecting tasks adaptively through IRT improved those figures to 0.048 and 0.059.

The advantage was not universal. With few reference systems and 20 executed tasks, IRT outperformed the combination in one investigated setting. For head-to-head rankings on Terminal-Bench, IRT was the most accurate estimator at every tested budget. Better score estimation therefore did not consistently translate into better pairwise ranking.

Costs and limits

The paper estimates that reaching error below 0.10 on SWE-bench could require $2 in target execution costs with the combined method, versus $10 for IRT and $40 for the observed sample score. These are conditional estimates: they assume $2 per execution and negligible rubric-generation, summarization and embedding costs. They are not an all-in deployment-cost measurement.

The experiments also use curated reference collections with nearly complete coverage of a single benchmark. Sparse, disjoint and cross-benchmark collections were not evaluated. Only DeepSeek-V4-Flash generated rubrics and summaries, leaving sensitivity to that model choice untested.

The preprint supports a specific finding: cached execution behavior can supplement correctness outcomes when predicting benchmark scores from few tasks. It does not establish effectiveness for online monitoring, agent steering or real-world deployment safety.