Teaching a Vision-Language Model When to Abstain—While Preserving Its Answers

A trade-off in saying “none of the above”

A vision-language model answers questions about images. In this study, some questions were made unanswerable in specific ways: the correct choice was missing, the answer set did not fit the question, or the image was incompatible with it. The task marked choosing the displayed “none of the above” option as correct for these cases. On ordinary questions with a valid answer, choosing that option counted as an error. These are benchmark definitions; they do not by themselves establish what a model should do in every real-world situation.

Jinyuan Zhang’s preprint, Multimodal Abstention That Preserves Acquired Correct Answers, examines a practical concern: adding abstention after a model has learned a task might undo some answers it learned to get right. The paper calls this “abstention repair.” Its objective is to increase correct abstention on unanswerable questions while limiting newly lost correct answers.

Overall accuracy can conceal that trade-off: lost correct answers and newly fixed errors may cancel out. The researchers therefore count losses and corrections separately and track whether answers gained through fine-tuning remain correct.

Score, calibrate, veto

The proposed method, ReFrain, leaves the answer model’s parameters unchanged. A linear probe reads the model’s first-token logits—the scores assigned to possible next tokens—to distinguish Standard records from the study’s unanswerable-problem-detection records. Those unanswerable records include the missing-answer, incompatible-answer-set and incompatible-image cases. The probe is trained on labeled examples.

Next, a threshold is selected using a held-out calibration set grouped into image clusters. A higher threshold can flag more questions for abstention, but may also reject answers the model previously got right. The researchers select a threshold to meet a specified budget for such losses on calibration data. At inference, ReFrain replaces a flagged response with the “none of the above” option displayed in the question; otherwise, it leaves the original response unchanged.

This one-way veto cannot correct a substantive wrong answer, but it also cannot rewrite one substantive answer as another. “Regression-free” refers to the zero-loss target on the calibration set—not a promise that future data will have no losses.

What the tests found

The main comparison began with the same answering-fine-tuned Qwen2.5-VL-3B checkpoint for each method. On the study’s evaluation set, ReFrain’s zero-budget setting retained all 57 answers newly corrected by fine-tuning and observed no losses among answers the checkpoint had got right. Correct abstention rose from 54.63% to 61.94%. This is an observed result on that evaluation set, not a general guarantee; the authors describe the evaluation population as historically exposed during development.

With a 0.5% calibration budget, ReFrain reached 74.05% correct abstention. On the evaluation set it lost eight originally correct answers and retained 55 of the 57 fine-tuning gains. CLIP-UP EmbLoRA reached 64.65% correct abstention, with 62 losses, 41 corrections and 40 retained gains. These results compare methods under this study’s shared-checkpoint setup; they should not be read as a universal ranking.

Threshold choice mattered. The same probe at its default threshold reached 92.16% correct abstention but lost 298 originally correct answers. A higher abstention rate alone, then, does not show that a method makes a better trade-off.

The researchers also tested frozen models and thresholds on 228 independent question clusters built from AI2D images. At the 0.5% setting, ReFrain reached 66.05% correct abstention and observed no losses among 912 originally correct answers. It exceeded EmbLoRA by 32.72 percentage points in correct abstention on this set. Plain abstention fine-tuning achieved a higher rate than ReFrain at that setting, but lost some original correct answers. The methods offer different trade-offs.

What remains uncertain

The independent test is a single evaluation using science-diagram images and constructed unanswerable questions. Fixed rules and a multimodal model were used to review question and image transformations; the authors say two human annotators did not perform that review. Although the confirmation images were not used to develop the project, exposure to public AI2D data during the base model’s pretraining or instruction tuning cannot be excluded.

The main post-adaptation retention experiment used one training seed and a training set of 677 records. Its image-cluster intervals measure variation for fixed checkpoints, not the variability that could come from retraining. The authors also report post-adaptation retention results for only one model in the main setting. The calibration budget is empirical: satisfying it on calibration data does not ensure the same loss rate elsewhere. A stated statistical bound applies only to the zero-budget setting, under exchangeability assumptions, and concerns a single new image cluster—not a guarantee of zero losses across a whole evaluation set.

This is a preprint, so its claims should be treated as provisional rather than as established deployment guidance. It presents a useful evaluation question: how much abstention is gained, and which correct answers are put at risk? Tests across more models, prompts, image sources and real-world settings are needed to determine how broadly the results apply.