EvoML checks machine-learning pipelines in stages and retains successful repairs

Checking more than whether code runs

Building a machine-learning model involves preparing data, choosing a configuration, training, tuning and packaging the result. Errors in an early step can affect everything that follows, even when the software executes successfully. EvoML, described in a preprint by Zixiao Zhao and colleagues, combines reusable procedures with checks between major stages to address that problem.

The system takes a dataset path and a minimal description of the task. Three language-model agents divide the work: an Orchestrator controls progression, an Executor runs and adapts procedures, and a Curator updates the procedure library after a run. The default framework uses GPT-5.5 agents, with Codex available to adapt Python templates.

Four phases, nine reusable skills

EvoML organises work into task understanding, planning, training and post-training processing. Nine skills cover dataset preparation, statistical summaries, experiment planning, preprocessing, hyperparameter search, configuration promotion, full training, evaluation aggregation and artifact export. Each skill combines written instructions, Python templates and reference material describing expected outputs.

The procedures draw on nnU-Net, a self-configuring medical-image segmentation framework. EvoML uses dataset characteristics to guide configuration rather than generating every decision from scratch. It adds Optuna-based hyperparameter search and promotes a candidate configuration only when its improvement over the baseline exceeds a predefined threshold. Its full-training skill follows five-fold cross-validation with early stopping.

After each phase, the Orchestrator checks structure, execution and task suitability. Structural and execution checks are programmatic: required files must exist, formats must match expectations and processes must finish successfully. Semantic checks depend on the language model interpreting domain guidance. The paper illustrates this distinction with a preprocessing script that runs successfully but applies an MRI-style normalisation where its CT-processing reference calls for clipping.

Failed checks produce feedback for targeted retries. If the retry budget is exhausted, a minimal fallback must pass the same gate or the workflow stops. The authors report that normal recovery succeeded before this fallback was needed in their experiments. These gates restrict progression according to specified checks; they do not establish that every possible error will be detected.

Keeping repairs for later datasets

The Curator examines validation logs, retries and successful script adaptations after a run. It selects the skill with the highest accumulated retry count for inspection, then can add a recovered script and guidance explaining when to use it. The nine-skill structure stays fixed.

One example concerns reconstruction data stored in HDF5. A preparation skill initially assumes the NIfTI layout used for segmentation, requiring recovery. The Curator retains the HDF5-handling script and adds routing instructions. Before acceptance, a patch must pass validation without retries on the input that originally caused difficulty. That check demonstrates recovery on the triggering case, rather than comprehensive validation across all previous inputs.

What the benchmarks showed

The main evaluation covers eight medical-imaging datasets spanning six task types. Comparisons include AutoML-Agent, MLZero, M3Builder, Codex and Claude Code. Each method runs three times per dataset, with means and standard deviations reported. The automation measure rewards five completed milestones and penalises human interventions.

EvoML achieved the maximum automation score on every medical dataset and the highest reported model quality among those comparison systems. On FastMRI reconstruction, its structural similarity score was 0.9498, versus 0.8579 for the next-best method. However, the paper also documents different input–target construction choices across systems, including comparison pipelines that synthesised inputs from targets rather than using measured k-space.

Removing validation gates reduced FastMRI quality from 0.9498 to 0.7116. Disabling retained skill evolution, while preserving online adaptation, increased repeated recovery across a fixed dataset sequence. With evolution, synthesis time fell from roughly 490 to 290 minutes and token use from 71 million to 50 million. First-pass skill success after the initial dataset rose from 57.1% to 79.4%.

Promising results, bounded evidence

Additional experiments on CIFAR-10, VOC2012 and AG News found near-perfect automation across methods, with EvoML reporting the highest model-quality scores. These provide evidence beyond medical imaging, but only on three additional tasks.

Computation is an important qualification. For a typical supervised segmentation task, the authors report approximately 60 hours of EvoML training versus 10 hours for the comparison systems. Pipeline synthesis time is separate from that training time. EvoML also trails expert leaderboard references on some datasets, including ACDC and BTCV. The preprint therefore supports a promising approach to checked, reusable automation—not universal superiority or guaranteed pipeline correctness.