TRANSMISSION // 2609.33822v2

How Vestrum Teaches AI Agents to Improve Their Setup

Source published

Vestrum uses failures in a few long task runs to adjust parts of an AI agent’s software harness. Early benchmark results are encouraging, but the preprint leaves questions about repeatability and transfer.

AI-generated conceptual illustration for How Vestrum Teaches AI Agents to Improve Their Setup
AI-generated conceptual illustration; not documentary evidence.

An AI agent’s performance depends not only on its language model but also on the surrounding software: what information it retrieves, how it breaks down a task, what it remembers and when it checks its work. A preprint introduces Vestrum, a method for improving those components using a handful of long task episodes. Its reported gains span coding, memory and cell-type annotation benchmarks, though the experiments have important limits.

An AI agent is more than the language model that generates its words. The surrounding software—often called a harness—can decide which information the model sees, how a task is divided, what gets saved for later and whether an intermediate result is checked before use. If the harness makes a poor choice, an error early in a long task can affect what follows.

Many methods for improving agent systems search over software they control and learn from dozens or hundreds of tasks. Vestrum asks a different question: can researchers improve an existing harness, even one they do not own, from only a few long episodes that may each involve hundreds of tool calls?

Learning from failures, not just scores

The authors tested Vestrum in five benchmark settings. Two used Claude Code, a coding agent; three used General Agentic Memory (GAM), a system for storing and retrieving information. Development sets ranged from one conversation to ten episodes.

Vestrum examines records of what the agent did, rather than relying only on a final score. It tries to locate where a failure began and describe it as a recurring type of mistake—for example, changing code before establishing what it must preserve. It then proposes a targeted change to one part of the harness. The four areas it can modify are verification, retrieval, task decomposition and knowledge synthesis: what information gets organized and retained.

A filter is intended to prevent changes from copying answers or details from development tasks. It combines a language-model review with checks for task-specific names, paths, symbols and constants. Candidate changes are tested against the current harness on development tasks and, in some settings, validation tasks. The selected harness is frozen before testing. The language model itself is not trained by this process.

Gains across varied tests

On Terminal-Bench 4 Hard, a coding benchmark, the Vestrum-modified Claude Code passed 70.3% of test checks, compared with 63.7% for the original harness. The test involved eight tasks, with two runs per harness; the modified version’s test cost was reported as 1.03 times the baseline’s.

On PathBench, which asks an agent to reproduce expert cell-type annotations from spatial transcriptomics data, agreement on held-out sections rose from 67.5% to 77.8%. The sections came from one slide, a limitation when judging how well the result might transfer to other samples.

On three memory benchmarks, the paper reports improvements over GAM: UltraHorizon rose from 47.6 to 59.8 on its score scale; AMA-Bench accuracy rose from 64.2% to 74.2%; and LoCoMo accuracy rose from 71.5% to 74.9%. The LoCoMo evaluation excludes its adversarial, unanswerable questions, so the result applies to the categories that remained.

The authors also compared Vestrum with GEPA, a prompt-search method, using the same development data. Vestrum scored higher in each paired evaluation. But the comparison does not hold the search space constant: GEPA adjusted prompts, while Vestrum could also change memory operations and control flow. The systems also used different task models during search.

Where checks helped—and where they hurt

The paper describes retained self-checks that acted before a result was committed: for example, comparing a memory note with its source evidence, or requiring an invariant and a test probe before the first code edit. These are observed design choices from the searches, not evidence that early checks improve outcomes in every setting. The Terminal-Bench edit gate was not measured at test. The paper’s test-time ablation of self-verification covered only UltraHorizon grid episodes and LoCoMo; switching checks off lowered scores in those evaluations.

By contrast, some reviewers that re-examined finished answers changed correct answers into incorrect ones. In development tests, one such reviewer fixed five answers but broke nine on AMA-Bench; on LoCoMo, another improved two and regressed six. Other verification designs also had mixed results. A check can block an answer without having enough evidence to identify the right one.

Promising, but still preliminary

The manuscript is a preprint marked “under review,” not a peer-reviewed publication. Each benchmark setting had only one Vestrum search. Repeated test runs can reveal execution or judging variability, but they do not show whether a fresh search would find a similarly effective harness. The authors tested each learned harness within the benchmark where it was developed, rather than transferring it to another benchmark of the same kind.

Some settings lacked a validation split, and PathBench’s development and test sections came from the same slide. The filter checks whether a proposed mechanism contains task-specific material; it does not establish that the mechanism will generalize to every task of the stated kind. GEPA is the only learned baseline comparison reported.

The results suggest that tracing errors to specific points in an agent’s workflow may help improve existing systems with limited development data. Whether the approach remains reliable across repeated searches, new benchmarks and broader real-world tasks remains an open question.

Reviewed cross-references

These connections were reviewed by a model, not independently fact-checked by a human. Similarity does not imply agreement; consult the original papers and the evidence below.

  • Banking AI compliance depends on the agent harness, not just the model

    Supporting evidence: Both studies find that the harness can affect agent outcomes: Vestrum reports gains after modifying existing harness components on held-out benchmark tasks, while the banking study varies harnesses with the model held fixed and finds compliance differences that vary by model. This supports the bounded point that harness choices can matter, not that the effects are comparable in size or generalize across domains; Vestrum evaluates task performance, whereas the banking study measures simulated conduct-rule compliance.

    Model review: · openai-codex/gpt-6-luna

    Evidence and provenance

    Selected excerpts, not complete papers. PDF extraction can omit figures or damage equations, tables and reading order; sources may be preprints.

    • 2609.33822v2 — Original paper

      Excerpt reference: static/papers/2609.33822v2.md:60-107
      Basis: Paper text
      Source SHA-256: 032134f4ef074d05ab47936605191fc197a50af3a87ff814e70548afe017e64e

    • 2611.06116v1 — Original paper

      Excerpt reference: static/papers/2611.06116v1.md:13-50
      Basis: Paper text
      Source SHA-256: f76bae923aeae7fc88aeeef2fe3f66e7a4fe700a18b805fc74861f1f598dcbb1

    • 2611.06116v1 — Original paper

      Excerpt reference: static/papers/2611.06116v1.md:655-664
      Basis: Paper text
      Source SHA-256: f76bae923aeae7fc88aeeef2fe3f66e7a4fe700a18b805fc74861f1f598dcbb1

Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.