RSI-Dojo trains AI agents to improve other language models

A preprint by Shuzheng Gao and colleagues investigates whether language-model agents can learn to conduct post-training—the additional training used to improve a pretrained model’s capabilities. Rather than only giving an existing agent better instructions or tools, the researchers fine-tune the agent itself on demonstrations of selecting data, running experiments and responding to results.

The distinction matters. An agent’s surrounding software, or harness, can guide its actions without changing the model’s learned behavior. RSI-Dojo combines improved guidance during demonstration collection with subsequent training intended to make useful model-training practices part of the agent’s capabilities.

Turning research into practice environments

The foundation is RSI-Craft, a pipeline that converts research datasets and evaluation tasks into executable environments. It draws candidates from recent papers at seven research venues and from the InspectAI evaluation platform. Agents check whether candidates provide reusable data or executable evaluation targets, then locate datasets, reproduce evaluation dependencies and standardize the interface.

The pipeline starts with 547 candidate tasks and produces 315 runnable environments before further filtering. Tasks with fewer than 50 examples are discarded; those with more than 300 are randomly reduced to 300 to limit evaluation time.

Further checks remove unreliable evaluations and tasks that do not sufficiently distinguish two reference models. The final collection contains 281 environments across 10 domains. This is a deliberately selected training collection, not an unrestricted sample of every kind of model-improvement problem.

Learning from successful training sessions

The researchers use frontier models to collect demonstrations in these environments. Even these models make recurring mistakes: training on too little data, changing recipes without examining failures, mishandling end-of-sequence tokens or failing to submit a usable final model. Incorrect end-of-sequence settings can leave a trained model unable to stop generating properly.

To address these patterns, the collection harness adds guidance on diagnosis, data selection, configuration and submission. The researchers then filter demonstrations for both process quality and outcomes. They remove problematic behavior, detected cheating and sessions whose final model does not measurably improve on its target task.

The retained material comprises 324 command-line sessions, split into 1,240 training trajectories at context-compaction boundaries. These examples are used to fine-tune Qwen3.5-9B and Qwen3.6-35B-A3B, producing the two RSI-Dojo agents.

What the benchmark results show

The main evaluation uses PostTrainBench v1.1. Each task gives an agent a pretrained target model, an evaluation interface, one GPU and a 10-hour budget. Its job is to improve the target model through data preparation, training and iteration. A smaller evaluation, PostTrainBench-Lite, uses 12 model-and-benchmark combinations and a five-hour budget.

On the full benchmark, the authors report that the smaller agent’s score rises from 7.72 to 16.46 after fine-tuning. The larger agent improves from 13.08 to 29.95. These are gains of 8.74 and 16.87 benchmark points, respectively—not percentage improvements in general intelligence.

The larger agent also becomes more reliable at executing training. It completes at least one successful training run in all 28 full-benchmark settings, compared with 15 before fine-tuning. Its final submitted models improve over their corresponding pretrained targets in 26 settings, up from 11.

In the authors’ evaluations, its full-benchmark score is slightly above GLM-5.2 and GPT-5.5, but below DeepSeek-V4.1-Flash. On Lite, it scores 53.29, close to GPT-5.5’s 52.96 and below the other two models. These comparisons concern automated post-training under the reported conditions, not overall model capability.

An ablation supports using both demonstration filters: the larger agent scores 53.29 on Lite with combined filtering, versus 30.66 with behavior-only filtering and 26.13 with outcome-only filtering.

A step toward automation, not a closed self-improvement loop

The evaluation has important qualifications. The researchers use a less powerful GPU than the official benchmark specifies. Their self-hosted agents lack the harness’s custom web-search tool, although they retain general network access and receive pre-downloaded training resources. Additional guidance used to collect demonstrations is not included during evaluation.

Performance also remains uneven. The authors identify limited successful training demonstrations for difficult mathematical reasoning and an absence of teacher-distillation trajectories as gaps affecting particular benchmarks.

Most importantly, the paper explicitly stops short of recursive self-improvement. The trained agents improve smaller target models; they are not shown training improved successors of themselves. The evidence supports a narrower conclusion: filtered demonstrations can improve agents’ ability to execute and refine bounded post-training workflows. Whether that capability extends to longer research projects or sustained self-improvement remains outside the demonstrated results.