Can an optimizer help AI learn new tasks without forgetting old ones?

A model trained on one task at a time faces a balancing act: it must adapt to new information while keeping useful earlier capabilities. This is called continual learning. If updates for a new task shift the model too far from a configuration that supported earlier tasks, its old predictions may deteriorate—a problem known as catastrophic forgetting.

The authors of STABLE: A Continual Learning Optimizer with Adaptive Drift Control focus on what happens just after a task switch. Their proposal is to control the training process itself, rather than relying solely on replaying past examples, adding task-specific components or applying a fixed penalty against parameter changes.

A flexible brake on parameter drift

The method, called STABLE, treats the parameters being trained as a point that can move through a high-dimensional space. At the end of each task, it records the current parameter values as an anchor. During the next task, it tracks how far the parameters move from that anchor and compares the distance with a drift budget.

STABLE uses a primal-dual optimizer: in practical terms, it adjusts the strength of a pull back toward the anchor as training unfolds. When movement exceeds the budget, the pull strengthens; when movement falls back below it, the pull weakens. The aim is to restrain abrupt changes near task boundaries without permanently preventing the model from learning the new task. The budget is calibrated using the first task, and the anchor is updated at each subsequent boundary.

The paper also gives a mathematical argument for why limiting this movement could help. For distance-based classifiers, it proves that parameter drift bounds the erosion of the model’s prediction margin, provided a local score-stability condition holds. A margin is the gap between the score for the correct class and the strongest competing class. The guarantee is conditional: it applies within the stated setting and neighborhood, not automatically to every model or learning objective.

Tests on image and vision-language tasks

The experiments use pretrained Vision Transformer and CLIP backbones, with training restricted to parameter-efficient additions called LoRA and prompts. The researchers compare these standard configurations trained with STABLE against versions trained with Adam, as well as against specialized continual-learning approaches.

For image classification, they test sequential streams drawn from ImageNet-R, CIFAR-100 and a 100-class subset of ImageNet-A. The streams contain 50 or 100 tasks, with two classes introduced per task. In this setting, standard LoRA trained with Adam performed very poorly on the longer streams, while replacing Adam with STABLE improved final average accuracy and reduced forgetting. For example, on the 100-task ImageNet-R stream, the reported final average accuracy was 73.12% with STABLE, compared with 0.66% for vanilla LoRA with Adam. Prompt tuning with STABLE also performed competitively with specialized prompt-based methods.

In CLIP experiments, the authors tested 10-task streams and measured both performance on the incoming classes and zero-shot transfer to separate image datasets. STABLE improved in-stream results over vanilla LoRA with Adam on all three streams tested. It also kept zero-shot performance close to that of pretrained CLIP in the reported comparisons. On the ImageNet-R stream, for instance, the STABLE model scored 88.56% on Food-101, compared with 88.38% for pretrained CLIP; these figures describe this particular evaluation, not a general improvement in zero-shot ability.

The researchers’ analyses support their proposed mechanism: after task switches, STABLE’s control variable increased when drift exceeded its budget and relaxed as the model returned within it. In selected experiments, measured margin erosion tracked forgetting. These observations are consistent with the paper’s explanation, but they do not establish that parameter drift is the only source of forgetting.

What remains uncertain

The scope is narrower than the broad problem of continual learning. The experiments use pretrained vision and vision-language models with parameter-efficient tuning; they do not test training from scratch, full-model fine-tuning, generative tasks or other modalities. The theoretical result is for representation-based distance classifiers, including the heads used in the study. The authors identify broader heads and objectives as future work.

Comparisons also require care. Some specialized methods use different inference procedures, although the paper includes a common-head comparison in an appendix. The results are reported across three random seeds, and performance on benchmark task streams cannot guarantee the same behavior in real-world systems facing different data, task sequences or operational constraints. STABLE is a training technique, not evidence that a model will reliably retain every capability after every update.

The paper states that it was accepted to NeurIPS 2026 and is to appear in the proceedings. This article is based on the supplied paper text; the work should still be read with its evaluation scope and assumptions in mind. The authors report code availability, which may help others reproduce and extend the experiments.