Measuring Representation Drift to Help Vision Models Learn Without Forgetting

Learning from a changing visual world creates a difficult trade-off: adapt too freely and earlier knowledge deteriorates; constrain learning too tightly and new domains remain unfamiliar. In domain-incremental learning, the categories stay the same while their appearance changes across successive datasets. The model must retain performance without revisiting earlier training images.

Hyunmin Kong, Gyutae Oh and Jitae Shin of Sungkyunkwan University address this problem in Drift-adaptive Representation Preservation for Domain-Incremental Learning. The paper’s arXiv version was submitted on November 4, 2026, and its landing page reports acceptance to ACCV 2026. It is therefore an arXiv manuscript reporting conference-accepted research, not merely a submission with no stated acceptance.

A drift meter for individual concepts

The authors’ starting observation is that forgetting varies across domains and classes, rather than following training order alone. Their method measures representation drift separately for each domain–class pair: how far a reconstructed representation has moved from a reference for its class. That distance is a surrogate for forgetting risk, not a direct measurement of lost accuracy.

The system uses a frozen DINOv2 ViT-B/14 vision backbone. Its attention weights produce two overlapping, softly weighted summaries of an image’s patch features. One emphasizes regions receiving stronger attention, intended to capture class-relevant evidence. The other uses complementary weights, tending to capture domain-sensitive information such as background, texture and style. These are not hard object-versus-background masks.

A small trainable encoder processes both summaries, and a predictor learns to reconstruct the target representation from the context. Inspired by Joint-Embedding Predictive Architectures, this prediction operates in feature space rather than generating images. A slowly updated teacher encoder supplies a stabilizing target, mixed with the labeled class’s prototype. Training combines classification, prediction and preservation losses; this is not a purely self-supervised procedure.

After each domain, the method stores two feature vectors for each observed class: a mean target representation and a mean context representation. It retains no past raw images. During later training, stored contexts pass through the current encoder and predictor, allowing the system to reassess earlier concepts.

Immediately before learning a new domain, the method fixes a reference anchor for each class using a snapshot of the teacher and the existing class prototype. Anchors remain fixed during that domain’s training, rather than remaining unchanged throughout the entire sequence. Drift is the cosine distance between a reconstructed past concept and its class anchor.

One signal, two different responses

High drift triggers stronger preservation pressure. The consolidation loss gives more weight to past domain–class pairs that have moved farther from their anchors, concentrating effort where the model detects greater change.

Prototype construction uses the same signal in the opposite direction. Classification compares an image representation with class prototypes assembled from stored domain-specific target means. Highly drifted domains receive less influence in that assembly, reducing what the authors call prototype contamination. Weight limits prevent individual contributions from becoming arbitrarily dominant.

The distinction matters: a drifting concept receives more help during training but less trust when constructing the classification reference. Only the lightweight adaptation components are trained; gradients do not pass through the backbone.

Competitive results, with trade-offs

Across three runs with different random seeds, the method achieved the highest reported average accuracy among compared methods on CORe50 and Office-Home: 99.7% and 87.8%, respectively. Reported average forgetting was 0.3% and 0.2%. Accuracy here means the average across domains after the final training stage; forgetting measures the decline from earlier peak performance on past domains.

On DomainNet, it reached 61.8% accuracy with 4.0% forgetting, below DUCT’s 66.4% accuracy. On CDDB-Hard, it reached 79.4% accuracy with 6.7% forgetting, below MoE’s 81.4% accuracy. The contribution is therefore a balance across benchmarks, not a sweep of every metric.

The DomainNet efficiency comparison reported 3.0 GB peak training GPU memory, versus 14.07 GB for DUCT and 32.0 GB for MoE. GraphKeeper used less, at 2.0 GB. Accumulated storage reached 13.78 MB after six domains, the smallest reported among compared methods.

Component-removal experiments support the preservation mechanism: in the DomainNet ablation table, removing consolidation increased forgetting from 4.12% to 14.50%. Removing either drift-aware weighting mechanism also worsened forgetting.

What the evidence establishes

The experiments cover four vision benchmarks with specified domain orders and a fixed label space, using one frozen backbone architecture. They do not establish performance when new categories arrive or across arbitrary deployment streams. Three datasets also use the authors’ per-domain, per-class train/validation/test re-splits, an important detail when comparing results with other protocols.

For continual-learning research, the central result is a practical way to turn compact memories into a feedback signal: measure which concepts are changing, protect them selectively, and prevent residual drift from distorting shared class references.