TRANSMISSION // 2611.23179v1
When AI says less, does it see better?
Source published
A study of LLaVA captioning models finds that lower hallucination scores can accompany fewer named objects. A recall-matched test separates brevity from other gains.

A caption that avoids naming the wrong objects may also leave out things that are really there. Hansang Lee’s preprint tests that trade-off in one family of vision-language models—and explores a training-free way to restore some missing object categories.
A vision-language model can make two different kinds of mistake when describing a picture: it can name an object that is not there, or fail to mention one that is. A lower hallucination score sounds like clear progress, but it can be misleading if the model simply says less. Hansang Lee’s study asks whether preference-optimized captioning models reduce hallucinations by becoming more selective—and how their results compare with a simpler change: making the original model stop earlier.
The paper examines LLaVA-1.5-7B and four publicly available versions fine-tuned using preference optimization, a family of methods that trains models to favor some answers over others. The author evaluates captions on CHAIR, a COCO-based benchmark for object hallucination, and also uses AMBER, a separate benchmark. The main comparisons include two disjoint sets of 500 COCO images. To account for how many real objects a caption names, the study measures object recall alongside hallucination rates. It also compares each fine-tuned model with the base model adjusted to reach the same recall.
That adjustment matters. Hallucination rates count errors among mentioned objects; a caption that names fewer objects can lower the rate without improving its overall accounting of the scene. In the study’s CHAIR-500 test, all four fine-tuned checkpoints lowered hallucination measures and object recall together. They named fewer distinct object categories per caption, even though their captions remained long. Set-level object F1—which combines how many objects were named with how many were correct—showed no statistically significant gain for any checkpoint on that set. On a separate held-out COCO set, CSR’s F1 was lower than the base model’s, with the reported interval excluding zero.
The author then tested the base model with a positive bias on its end-of-sequence, or EOS, token. In effect, this makes it more likely to end a caption earlier. At the recall of three preference-optimized checkpoints, this change alone reproduced 58–86% of their CHAIRi gain on CHAIR-500. The paper reports a similar pattern on the held-out set. This is a comparison of model outputs, not proof that preference training caused the recall loss: the author explicitly makes no causal claim about training effects.
Could the fine-tuned models recover the missing objects simply by writing longer? Under the tested greedy-decoding setup, adding a negative EOS bias recovered only 2–9% of the lost CHAIR recall for three checkpoints before a preset length-limit rule became relevant. Of the added object mentions, 94–96% repeated categories already named in the caption. Meanwhile, the base model’s caption of the same image named about half of the categories those fine-tuned models missed on CHAIR.
That observation motivates a training-free restoration method. The system takes object categories named by the base model but absent from the fine-tuned caption, ranks them using the base model’s token confidence, and appends those above a threshold. The threshold was set on development data and then applied unchanged to a held-out COCO set. For three checkpoints, restored captions came within one percentage point of the base model’s recall and had lower distinct-object hallucination rates than the base model. But the additions were far from reliably correct: only about 41–45% of appended categories were in the held-out image’s ground truth. The method is a benchmark intervention, not an image-verification step or a guarantee that a restored caption is accurate.
The results also depend on what is being compared. Against the base model’s stopping curve at matched recall, restoration showed an advantage for RLAIF-V near the base model’s recall, and for RLAIF-V and CHAIR-DPO at an intermediate recall. It did not show that advantage for CSR near base recall. The stopping curve itself covers one specific policy—changing the base model’s EOS bias—not every possible way to produce captions at a given recall.
There are further limits to the evidence. The study uses one model family and one backbone, greedy decoding, a detailed-caption prompt and two non-overlapping image sets from the same COCO dataset. Its results concern benchmark-recognized objects, not every kind of visual error, such as mistakes about attributes or relationships. Reported intervals are nominal 95% intervals, with no correction for multiple comparisons. The paper is an arXiv preprint, and the source record does not establish that it has undergone peer review. The findings are therefore best read as a careful diagnosis of a particular model-and-benchmark setup—not a verdict on all preference optimization or image-captioning systems.
- vision-language models
- object hallucination
- object recall
- preference optimization
- LLaVA
- image captioning
Related reading
Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.
An image-caption evaluator adds uncertainty to its scores
Source published
Shared keywords: image captioning, vision language models
MMCOMET gives commonsense knowledge a visual anchor
Source published
Shared keywords: vision language models
RECAP speeds up vision-language models by recycling predictions in context
Source published
Shared keywords: vision language models
A More Targeted Way to Make Vision–Language AI Resist Image Attacks
Source published
Shared keywords: vision language models
Can Rewritten Captions Make an AI Stop Recognizing a Face?
Source published
Shared keywords: vision language models