A model’s first self-update may not tell the whole story

A model update is often judged by comparing performance before and after training. But in a self-updating loop, the new model may generate the material used for its next update. The first update therefore changes not only the model being tested, but also the starting point for the next round. Third Eye: Beyond Immediate Gains in Recursive Self-Improvement asks whether an update’s measured value changes after one further update.

The researchers tested five language models, ranging from 3 billion to 8 billion parameters, on math and code. Starting from released model checkpoints, they made small training batches from problems the models had initially failed. The models generated corrections, and only corrections that passed an answer check or code execution test were kept. Each update used two corrections and LoRA, a method that adjusts a limited set of added parameters rather than retraining the entire model.

For each experiment, the team made three candidate updates from the same starting model. Separate development questions were used to select a candidate; different final questions were used to score it. Then each candidate generated new correction data and underwent one further update, with three continuation draws per candidate. The researchers compared both stages with the unchanged starting model. Their utility measure gave equal weight to three categories: the task used to make corrections, a different set of problems to test transfer, and a general-knowledge set used to track retained abilities.

Across the study there were 64 fresh experimental runs, 192 candidates, and 576 continuation draws. These counts do not mean there were hundreds of independent models: experiments reused five released starting checkpoints, and repeated measurements within a run are related.

The clearest comparison combined Qwen3-4B and Llama-3.2-3B. After candidates were selected using development scores, their average utility after the first update was essentially zero: +0.016 percentage points. After continuation, it was −1.760 points. The paired change was −1.776 points, with a simultaneous 95% interval from −2.876 to −0.679. In this comparison, measured utility was lower after the further update. That result does not establish what would happen with every model or training setup.

Candidate selection matters. When the researchers averaged all three candidates instead of selecting the best by development performance, immediate utility was already negative, at −0.686 points. Selection therefore helped make the first-step result look neutral; it did not remove the later decline.

Results varied across the other models. Continuation also reduced measured utility for SmolLM3-3B and Llama-3.1-8B. For Qwen3-8B, the estimated change was −0.681 points, but its adjusted interval included both a decline and a modest improvement, so the result is inconclusive. The pattern does not show that every model gets worse, or that model size determines the outcome.

The losses were not simply a story of forgetting general knowledge. In models with the largest declines, changes on the target task and on out-of-distribution problems tended to be larger than changes on the retention measure. The study also found more answer-format or code-syntax failures in some settings, but cannot establish how much these explain the score changes. Matched controls that varied training steps, correction batches, and which model supplied corrections did not identify a reliable fix or a single mechanism. Some control estimates were imprecise; the authors caution that these results do not rule out alternatives.

The practical message is about evaluation: report the first update, the outcome after a further update, and a no-update baseline, using independent questions for selection and final scoring. An immediate score is informative, but cannot stand in for how a model behaves after continuing to learn.

There are important limits. This is an arXiv preprint, and the supplied source does not establish its peer-review status. The experiments cover five checkpoints, two tasks, small two-example updates, and one further update. Longer learning loops, larger training batches, and full fine-tuning were not tested. Runs from the same checkpoint are not independent pretraining experiments, and the authors note limits in test coverage and uncertainty estimates. The work measures outcomes under a particular protocol; it does not show that self-improvement generally fails, explain the cause of the observed losses, or demonstrate a dependable way to predict which future updates will help.