TRANSMISSION // 2609.33822v2
How Vestrum Teaches AI Agents to Improve Their Setup
Source published
Vestrum uses failures in a few long task runs to adjust parts of an AI agent’s software harness. Early benchmark results are encouraging, but the preprint leaves questions about repeatability and transfer.

An AI agent’s performance depends not only on its language model but also on the surrounding software: what information it retrieves, how it breaks down a task, what it remembers and when it checks its work. A preprint introduces Vestrum, a method for improving those components using a handful of long task episodes. Its reported gains span coding, memory and cell-type annotation benchmarks, though the experiments have important limits.
An AI agent is more than the language model that generates its words. The surrounding software—often called a harness—can decide which information the model sees, how a task is divided, what gets saved for later and whether an intermediate result is checked before use. If the harness makes a poor choice, an error early in a long task can affect what follows.
Many methods for improving agent systems search over software they control and learn from dozens or hundreds of tasks. Vestrum asks a different question: can researchers improve an existing harness, even one they do not own, from only a few long episodes that may each involve hundreds of tool calls?
Learning from failures, not just scores
The authors tested Vestrum in five benchmark settings. Two used Claude Code, a coding agent; three used General Agentic Memory (GAM), a system for storing and retrieving information. Development sets ranged from one conversation to ten episodes.
Vestrum examines records of what the agent did, rather than relying only on a final score. It tries to locate where a failure began and describe it as a recurring type of mistake—for example, changing code before establishing what it must preserve. It then proposes a targeted change to one part of the harness. The four areas it can modify are verification, retrieval, task decomposition and knowledge synthesis: what information gets organized and retained.
A filter is intended to prevent changes from copying answers or details from development tasks. It combines a language-model review with checks for task-specific names, paths, symbols and constants. Candidate changes are tested against the current harness on development tasks and, in some settings, validation tasks. The selected harness is frozen before testing. The language model itself is not trained by this process.
Gains across varied tests
On Terminal-Bench 4 Hard, a coding benchmark, the Vestrum-modified Claude Code passed 70.3% of test checks, compared with 63.7% for the original harness. The test involved eight tasks, with two runs per harness; the modified version’s test cost was reported as 1.03 times the baseline’s.
On PathBench, which asks an agent to reproduce expert cell-type annotations from spatial transcriptomics data, agreement on held-out sections rose from 67.5% to 77.8%. The sections came from one slide, a limitation when judging how well the result might transfer to other samples.
On three memory benchmarks, the paper reports improvements over GAM: UltraHorizon rose from 47.6 to 59.8 on its score scale; AMA-Bench accuracy rose from 64.2% to 74.2%; and LoCoMo accuracy rose from 71.5% to 74.9%. The LoCoMo evaluation excludes its adversarial, unanswerable questions, so the result applies to the categories that remained.
The authors also compared Vestrum with GEPA, a prompt-search method, using the same development data. Vestrum scored higher in each paired evaluation. But the comparison does not hold the search space constant: GEPA adjusted prompts, while Vestrum could also change memory operations and control flow. The systems also used different task models during search.
Where checks helped—and where they hurt
The paper describes retained self-checks that acted before a result was committed: for example, comparing a memory note with its source evidence, or requiring an invariant and a test probe before the first code edit. These are observed design choices from the searches, not evidence that early checks improve outcomes in every setting. The Terminal-Bench edit gate was not measured at test. The paper’s test-time ablation of self-verification covered only UltraHorizon grid episodes and LoCoMo; switching checks off lowered scores in those evaluations.
By contrast, some reviewers that re-examined finished answers changed correct answers into incorrect ones. In development tests, one such reviewer fixed five answers but broke nine on AMA-Bench; on LoCoMo, another improved two and regressed six. Other verification designs also had mixed results. A check can block an answer without having enough evidence to identify the right one.
Promising, but still preliminary
The manuscript is a preprint marked “under review,” not a peer-reviewed publication. Each benchmark setting had only one Vestrum search. Repeated test runs can reveal execution or judging variability, but they do not show whether a fresh search would find a similarly effective harness. The authors tested each learned harness within the benchmark where it was developed, rather than transferring it to another benchmark of the same kind.
Some settings lacked a validation split, and PathBench’s development and test sections came from the same slide. The filter checks whether a proposed mechanism contains task-specific material; it does not establish that the mechanism will generalize to every task of the stated kind. GEPA is the only learned baseline comparison reported.
The results suggest that tracing errors to specific points in an agent’s workflow may help improve existing systems with limited development data. Whether the approach remains reliable across repeated searches, new benchmarks and broader real-world tasks remains an open question.
- AI agents
- agent harnesses
- Vestrum
- AI memory
- verification
- long-horizon tasks
- preprint
Reviewed cross-references
These connections were reviewed by a model, not independently fact-checked by a human. Similarity does not imply agreement; consult the original papers and the evidence below.
Banking AI compliance depends on the agent harness, not just the model
Supporting evidence: Both studies find that the harness can affect agent outcomes: Vestrum reports gains after modifying existing harness components on held-out benchmark tasks, while the banking study varies harnesses with the model held fixed and finds compliance differences that vary by model. This supports the bounded point that harness choices can matter, not that the effects are comparable in size or generalize across domains; Vestrum evaluates task performance, whereas the banking study measures simulated conduct-rule compliance.
Model review: · openai-codex/gpt-6-luna
Evidence and provenance
Selected excerpts, not complete papers. PDF extraction can omit figures or damage equations, tables and reading order; sources may be preprints.
2609.33822v2 — Original paper
Excerpt reference:
static/papers/2609.33822v2.md:60-107
Basis: Paper text
Source SHA-256:032134f4ef074d05ab47936605191fc197a50af3a87ff814e70548afe017e64e2611.06116v1 — Original paper
Excerpt reference:
static/papers/2611.06116v1.md:13-50
Basis: Paper text
Source SHA-256:f76bae923aeae7fc88aeeef2fe3f66e7a4fe700a18b805fc74861f1f598dcbb12611.06116v1 — Original paper
Excerpt reference:
static/papers/2611.06116v1.md:655-664
Basis: Paper text
Source SHA-256:f76bae923aeae7fc88aeeef2fe3f66e7a4fe700a18b805fc74861f1f598dcbb1
Related reading
Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.
MemSidecar puts AI memory retrieval on a separate track
Source published
Shared keywords: ai agents, long horizon tasks
A proposal to help AI experts draw on their creators’ experience
Source published
Shared keywords: ai agents, ai memory
Veritas gives AI-written task checkers a clearer view of the evidence
Source published
Shared keywords: ai agents
Can routers prioritize AI-agent traffic without trusting every label?
Source published
Shared keywords: ai agents
A Study Tests Two Ways for AI Assistants to Remember
Source published
Shared keywords: ai agents