Keeping reinforcement-learning agents able to learn

When better training tasks are not enough

Reinforcement-learning agents learn by taking actions and receiving rewards. In unsupervised environment design, software generates or selects the environments they encounter. Some methods adjust that selection using the agent’s performance, creating an adaptive curriculum.

But useful tasks are only part of the problem. Does the agent remain capable of learning from them as training continues?

In the preprint Training Stability under Adaptive Curricula, submitted to arXiv on November 5, 2026, Johan Obando-Ceron, Walter Mayor and colleagues investigate that question. Their experiments suggest that deterioration inside the learner can limit what an existing curriculum achieves. Addressing selected training problems improves performance without increasing the number of environment interactions within each comparison.

Examining the learner, not just its score

The researchers used Kinetix, a simulator containing procedurally generated physics-control tasks. Its Small, Medium and Large configurations permit different numbers of shapes, joints and thrusters, providing increasingly complex environments.

Agents learned with Proximal Policy Optimization, or PPO. Most experiments used two billion environment interactions and five random seeds. The authors reported interquartile means, which reduce the influence of extreme results, with bootstrap confidence intervals.

Their detailed diagnosis focused on Prioritized Level Replay, or PLR, which uses student-dependent scores to select stored levels for replay. They compared it with domain randomization, which samples environments from a fixed generator distribution.

PLR achieved higher sampled return while also showing less favorable internal diagnostics. These included larger gradients, more similar internal representations of different states, and weaker agreement between the ordering of predicted values and observed outcomes. Better task performance could therefore coexist with signs of training strain.

Within PLR, more complex environments were associated with stronger parameter growth, weaker value ordering and declining representational diversity. Highly prioritized levels could also have low measured learnability. Here, learnability is a success-rate-based proxy that peaks when a task is solved half the time—not a direct measurement of future improvement.

These patterns were associations, not proof that adaptive replay caused deterioration or that any single diagnostic explained poor performance.

Two changes to value learning

The team screened several established interventions, then concentrated on weight clipping and HL-Gauss.

Weight clipping constrains the critic’s parameters after updates. The critic is the component that estimates future rewards, helping guide the agent’s learning. Limiting parameter magnitude does not itself guarantee small gradients or small changes in behavior.

HL-Gauss changes how that critic learns. Instead of directly minimizing squared errors between scalar predictions and targets, it predicts probabilities across ordered value bins and learns against a smoothed target. This removes the direct dependence of the prediction-head gradient on the raw scalar error. It still estimates an expected value; it does not model the full distribution of possible returns.

Both interventions improved performance individually in the tested settings. Weight clipping was particularly effective in the most complex configuration. Their combination performed strongly across complexity levels, but did not consistently outperform clipping alone. The authors selected it as a practical configuration, not a proven optimal recipe or a universally synergistic combination.

The combined intervention controlled parameter growth and maintained stronger value–outcome ordering. It also reduced late-training representation deterioration in harder settings. However, higher representation rank was not always associated with better performance, cautioning against treating any one internal metric as a definitive measure of learning health.

Longer learning and unfamiliar tasks

Benefits extended beyond PLR to curricula based on estimated learnability or environment editing, and to domain randomization.

When PLR training on Small environments increased from two billion to five billion interactions, baseline return declined. The stabilized learner continued improving return while maintaining solve rate and stronger internal diagnostics. Stabilization also helped with limited parallel data collection.

Agents performed better without further adaptation on unfamiliar tasks across complexity levels. In a separate Small–Medium–Small training sequence, the stabilized learner retained more performance on Medium holdout tasks after training returned to Small.

An important distinction remains: the curriculum rule stayed unchanged, but the actual training distribution could change. A different learner produces different scores, which influence subsequent task selection.

What the evidence leaves open

The evidence comes from one simulator and one learning algorithm. Hyperparameter choices, diagnostic probes and interactions between interventions limit causal conclusions. The sequential experiment measured only Medium holdout performance, so it cannot fully separate forgetting, interference and reduced adaptability.

The preprint’s central lesson is therefore bounded but useful: evaluating a curriculum requires checking not only which experiences it supplies, but whether its learner can continue benefiting from them.