Can AI help improve a training run too expensive to repeat?

Improving the process, not an autonomous model

Can AI improve the machinery used to create another AI when a complete experiment is too expensive to repeat? A preprint from researchers at Shanghai Innovation Institute and Xinxu AI explores that question through the development of a language model called 801-30B-A3B.

The team reports training it from scratch on approximately 11.8 trillion tokens—pieces of text—in three months. It uses a mixture-of-experts architecture: around 30 billion parameters in total, with roughly three billion active for each token. Parameters are the numerical settings learned during training.

The researchers describe their approach as recursive self-improvement, but the definition matters. The improving system is the research project, including people, AI agents and infrastructure. It is not a model independently rewriting itself or taking over its own development. This is an empirical engineering report, not a theoretical demonstration of unlimited self-improvement, and it has not established its claims through peer review.

Smaller loops around one expensive run

A conventional improvement loop proposes a change, runs an experiment, measures the result and retains what worked. Full-scale pre-training makes that difficult: experiments are costly, and a promising benchmark score may not reveal how useful the resulting model will become after further training.

The team instead organized smaller loops around debugging, evaluating learning potential and improving data-processing components. Shared records connected training measurements, software changes, data mixtures and earlier investigations, including unsuccessful fixes.

For infrastructure debugging, agents searched across more than 130 code versions and 106 configuration fields. One investigation concerned how training work was distributed among the model’s experts. Overall measurements looked comparatively normal while individual layers developed substantial imbalances. Revised settings reduced an early imbalance, although they did not eliminate it throughout training.

Data investigations started from unwanted outputs, including repetitive configuration blocks and stereotyped question-and-answer templates. Agents searched 1,053 data subsets and traced suspicious material through processing stages. Humans prioritized investigations and approved fixes. This was supervised diagnosis, not unrestricted autonomous intervention.

Measuring what further training can unlock

Pre-training builds a base model; post-training subsequently adapts it to tasks and instructions. The authors argue that monitoring should assess the foundation’s potential for adaptation, rather than only its immediate performance.

Their main probe was supervised fine-tuning, or SFT: additional learning from example responses. A dataset of 6.33 billion tokens was small relative to the pre-training budget, although substantial in absolute terms. No reinforcement learning was used in this probe.

During one interval, direct base-model scores were nearly flat while the average after the same fine-tuning procedure rose from 70.54 to 72.93. Those averages came from different benchmark collections, so their absolute values cannot be directly compared. The informative observation is that the trends differed.

The team also fitted learning curves to measure improvement with additional fine-tuning data and tested alternative prompting formats. These are practical estimates of potential, not guarantees about what a future reinforcement-learning pipeline would achieve. Fine-tuning was not repeated with multiple random seeds, limiting confidence in small checkpoint differences.

Agents also produced training material

Another component turned software pull requests into executable tasks with instructions, environments and tests. The pipeline checked that an evaluation failed on the original code and passed on the reference solution, then removed the reference patch from the solver’s environment.

Approximately 220,000 executable tasks produced about two million trajectories across nine agent harnesses—the software interfaces through which models use tools. Filtering retained roughly 1.1 million trajectories, exceeding 100 billion tokens. Including other sources, the team assembled 134 billion tokens of agentic data available for pre-training; that figure is not a claim that every token entered the final run.

Checks targeted shortcuts such as fetching an existing fix or modifying tests. Such verification strengthens the dataset, but cannot establish that every retained trajectory genuinely solved its task without exploiting a loophole.

Competitive results, uncertain attribution

Across 13 base-model benchmarks, 801 averaged 73.13, behind Qwen3.5-35B-A3B at 76.67 and only narrowly ahead of Nemotron-3.5-Lightning at 73.07. That 0.06-point difference does not establish a robust advantage.

After lightweight SFT, it led the reported four-benchmark agentic average at 58.43, with a particularly strong KernelBench result. It did not win every individual benchmark. Comparisons against vendor-post-trained models also involved different training histories, rather than matched treatments.

The central limitation is attribution. Data, optimizer settings and infrastructure changed together, and there was no equivalent complete project without agents. The study therefore documents a competitive model built with AI-assisted workflows; it does not measure the agents’ isolated benefit. Its strongest lesson is organizational: smaller investigations and persistent shared evidence can support improvement even when the largest experiment cannot simply be rerun.