ACTOR uses delayed labels to improve later image classifications

Feedback for the next image

Vision-language models such as CLIP classify images by comparing their visual representations with text descriptions of possible classes. But performance can deteriorate when deployment images differ from those encountered during pretraining. Test-time adaptation tries to address that mismatch using incoming images, often relying on the model’s own confidence or predicted labels.

That creates a difficulty: an uncertain model may reinforce its mistakes rather than repair them. Jihwan Bang, Sumyeong Ahn, Hwanjun Song and Jae-Gil Lee investigate an alternative in Training-Free Active Test-Time Adaptation for Vision-Language Models. Their manuscript was posted to arXiv on November 6, 2026; the supplied landing record lists it as accepted at NeurIPS 2026.

The proposed setting allows a limited number of label requests, with a strict ordering rule. The system must issue its prediction before requesting the correct label. Returned feedback can help only with subsequent images. This prevents an evaluation from counting the requested answer as a successful prediction on the same image.

A memory with two checks

The researchers implement this setting through ACTOR, a training-free add-on that stores previously queried images and their returned labels. ACTOR itself requires no gradient-based optimization and leaves the underlying adaptation method unchanged. Some compatible methods, however, already perform their own prompt optimization.

For each incoming image, ACTOR first examines the base method’s predictive entropy: a measure of how spread out its class probabilities are. Low-uncertainty predictions pass through unchanged. Uncertain images become candidates for querying or memory-based correction.

After an initial memory-building phase, ACTOR retrieves the two nearest labeled images in the model’s visual representation space. Its main correction rule requires their labels to agree and their average cosine similarity to the current image to exceed a threshold. If both checks pass, their shared label replaces the base prediction.

Otherwise, ACTOR requests a label if budget remains, retaining the base prediction for the current image. Pending requests count against the budget but cannot enter usable memory until their labels return.

Once the budget is exhausted, the method retains its main correction rule but adds a looser fallback. If that rule fails, it accepts the nearest labeled image’s class only when that class appears among the base model’s five highest-ranked possibilities. This is not the same proximity safeguard as the main rule.

What the benchmarks showed

The main experiments used CLIP with ResNet-50 and ViT-B/16 image encoders. Evaluation covered ten cross-domain classification datasets, plus ImageNet and four ImageNet-derived distribution-shift datasets. The standard configuration allowed queries for up to 10% of the test stream, with labels returning one sample later. Results were averaged over three different stream orders.

With ViT-B/16, adding ACTOR increased zero-shot CLIP’s average accuracy across the ten cross-domain datasets from 61.94% to 70.67%. Added to the training-free Dynamic Adapter, or TDA, it raised that average from 68.14% to 72.66%.

The improvements were not universal at the individual-dataset level. For example, zero-shot CLIP’s ImageNet-V2 accuracy slipped from 59.80% to 59.69% with ACTOR. The broader finding is improved benchmark averages, not a guarantee that every dataset benefits.

To separate the retrieval method’s contribution from simple access to labels, the authors also compared it with two supervised prompt-tuning approaches using identical queried labels and future-only supervision. Across 15 datasets, CLIP with ACTOR averaged 67.65%, compared with 65.15% for Online-CoOp and 65.20% for Replay-CoOp.

Benefits with explicit costs

A controlled five-dataset comparison tested different ways of reusing the same labeled memory. ACTOR’s combined agreement-and-proximity rule achieved 69.50% average accuracy, versus 67.22% for five-neighbor majority voting. Harmful corrections—changes from a correct base prediction to an incorrect answer—accounted for 5.28% of ACTOR’s applied corrections, versus 13.68% for majority voting.

Measured computational overhead was below half a millisecond per image across the evaluated methods on the reported hardware. That figure excludes human annotation time and cost. The study used benchmark labels rather than conducting a human-annotation experiment.

Longer label delays reduced benefits, particularly when delays occupied a substantial fraction of a short stream. The authors also identify noisy feedback, uneven class coverage and reliance on the weaker fallback after budget exhaustion as limitations. ACTOR therefore demonstrates a way to reuse sparse, delayed supervision—not a label-free solution or evidence that real-world feedback is automatically reliable.