BoxBox lets coding agents revise their tools during software optimization

A November 6, 2026 preprint by Yuhao Tan and colleagues introduces BoxBox, a system for adapting a coding agent’s working machinery during an ongoing task. The researchers report better software performance improvements than three comparison methods under matched four-hour test budgets. The findings concern a specific benchmark and model, not general or unrestricted AI self-improvement.

What changes—and what does not

An agent’s harness is the software that presents tools, constructs model inputs, handles outputs and manages execution. BoxBox lets the agent revise this executable software, including guidance, command tools, context management and fault handling. The language model remains fixed: this is not retraining or modification of model weights.

When the agent requests a review, a separate supervisory kernel suspends its task-interaction loop and launches a review session. That session can inspect execution records and harness history, but treats project code as read-only. It may commit harness changes or leave the harness unchanged.

The kernel validates a candidate and attempts to activate it while transferring conversation state and retaining the existing terminal and workspace. Failed activation retains or restores the previous runtime. Successful revisions also govern later reviews, making the process recursive. Normal reviews preserve ongoing environment processes; runtime-fault recovery instead starts a fresh terminal, losing shell-local state while retaining project files and execution records.

Agent control has constraints: the initial harness requires intervening task activity and a 30-minute cooldown after a completed review.

Measuring useful optimization

The accompanying PerfRace benchmark contains 100 repository-level tasks across seven domains, including developer tools, data systems and scientific computing. Projects span 16 primary programming languages. Rather than reproducing a known optimization patch, agents receive an end-to-end workload and must discover bottlenecks themselves.

Construction began with 103,475 candidate projects identified through public GitHub coding-agent activity. Filtering selected 239 for agent–human verification, producing 100 accepted tasks. Reviewers assessed realistic use, measurement reliability and resistance to shortcuts. Accepted tasks required an average of 9.1 hours of curation, so this was not a fully automated construction process.

An external Judge evaluates snapshots in separate containers, using an independent workload copy and held-out correctness cases unavailable in the agent workspace. Verification consumes task time. Incorrect submissions cannot improve the score, and runs without a valid improvement retain the unmodified baseline score of 1×.

The principal outcome is best-so-far correct speedup, not necessarily the performance of the final workspace. Across three runs, each task’s results are combined geometrically; across tasks, the researchers use a harmonic mean to reduce the influence of extreme speedups.

Results and costs

The researchers allocated 40 tasks to offline training, 10 to validation and 50 to held-out testing. All methods started from the same terminal-based harness and used GPT-5.6 Luna with high reasoning effort. Each test method received three four-hour runs per task, including task execution, harness revision and verification.

Reported aggregate speedups and mean model costs per run were:

• Frozen Harness: 1.442×, costing $23.22.
• Harness Evolution: 1.281×, costing $24.56.
• Harness Scaling: 1.539×, costing $12.98.
• BoxBox: 1.702×, costing $16.43.

Frozen Harness never changes its harness. Harness Evolution learns revisions offline, then fixes the selected harness for testing. Harness Scaling uses an external controller to revise between attempts, preserving project changes but resetting conversation context.

BoxBox achieved the highest speedup, but Harness Scaling had the lowest model bill. BoxBox’s reported model cost was 29.3% below Frozen Harness. Its advantage on the paper’s speedup-per-dollar metric reflects an average of task-level ratios, not aggregate speedup divided by average cost. Including recorded offline training and validation raises Harness Evolution’s mean cost to at least $33.73 per test run.

What the revisions accomplished

BoxBox runs had a median of five reviews. Reviews consumed a median 2.1% of task time and 13.8% of model cost. Changes included batching experiments, waiting internally for commands instead of repeatedly invoking the model, and compacting long histories.

In one compiler task, reviews improved timing helpers and introduced alternating measurements of unchanged and modified versions. That comparison exposed an apparent improvement as measurement variation before later code changes produced verified gains.

Ablations on 14 test tasks supported both agent-triggered reviews and full-harness editing: full BoxBox scored 1.584×, versus 1.508× with scheduled reviews and 1.349× with restricted editing.

Limits and security

These preprint results use one model, a curated CPU-oriented benchmark and four-hour test budgets. Offline training used shorter runs, and the ablations covered only 14 tasks. Reported uncertainty bands resample runs within the fixed test set; they do not establish generalization to other projects or models.

Executable self-modification also introduces security risks. Repository content is untrusted, and revisions could weaken safeguards. The authors explicitly describe validation, rollback and isolated evaluation as incomplete security boundaries. BoxBox remains a research prototype, not a production-ready framework for autonomous self-modification.