An image-caption evaluator adds uncertainty to its scores

Beyond a single score

When an automated evaluator judges an image caption, it typically returns one number describing how closely the words match the picture. That number leaves an important question unanswered: how much trust should users place in the assessment?

In a preprint posted on November 6, 2026, Gonçalo Gomes, Bruno Martins and Chrysoula Zerva introduce BEiTScore++, an extension of their BEiTScore caption evaluator. It predicts both a central quality score and a range intended to capture uncertainty about the human rating. The results suggest that these ranges carry useful information about prediction errors, but do not consistently achieve their intended coverage.

The system is reference-free: it evaluates an image and a candidate caption directly, without needing a separate reference caption for comparison.

Learning a score and its surrounding range

BEiTScore++ uses the BEiT-3 image-and-language model, initialized from the original BEiTScore evaluator. The researchers trained it using human-rated image–caption pairs from VICR, Polaris, Nebula and EvalMuse-40K, normalizing ratings to a scale from zero to one.

Instead of predicting only one score, its output layer estimates a median and the boundaries of five nested prediction intervals, targeting coverage levels from 70% to 90%. A 90% interval is intended to contain the human target rating for roughly nine out of ten examples under suitable conditions. It is not a guarantee that any particular caption has been judged correctly.

The model constructs its boundaries outward from the median, keeping lower estimates below higher ones. This prevents “quantile crossing,” where independently predicted boundaries appear in the wrong order. Training also balances narrower intervals against penalties for missing human ratings.

After training, a separate calibration step adjusts the intervals using held-out examples. The researchers grouped images into 63 similarity clusters, calibrated within those groups, and selected the parameter pair producing the widest average interval. This favors coverage over precision: broader ranges can contain more ratings, but communicate less specific information.

Useful uncertainty, uneven coverage

The team evaluated agreement with human ratings on VICR, Polaris, Nebula, Flickr8K-Expert and Composite. The new model’s ranking agreement stayed close to that of BEiTScore, suggesting that adding uncertainty estimates did not substantially compromise its central scores.

For comparison, the researchers used Monte Carlo dropout, which repeatedly runs a model with some internal units randomly disabled and measures variation in its outputs. That baseline required 25 passes; BEiTScore++ produced its score and interval in one.

For both model sizes, the final BEiTScore++ configuration showed a stronger correlation between interval width and absolute prediction error on four of the five benchmarks. Polaris was the exception. In other words, wider intervals generally provided a better warning of larger errors than the baseline did, though not universally.

Actual coverage was less consistent than the nominal 90% target. For the final base-sized model, calibrated intervals contained human ratings in 90.5% of VICR examples and 97.6% of Polaris examples, but only 69.9% of Composite examples. The corresponding large model reached 86.8%, 97.4% and 63.7%.

These differences matter. Conformal calibration relies on assumptions about the relationship between calibration examples and future inputs; it does not ensure reliable coverage under arbitrary distribution shifts or for every subgroup. The clustering analysis also showed that its coverage gains mainly came from selecting wider intervals, rather than finer adaptation to image content.

Testing blurred images and unfamiliar artwork

To probe sensitivity to degraded inputs, the researchers blurred images from a rebalanced subset of 3,624 Polaris examples. Moderate blur increased interval widths. Under the strongest blur, however, widths decreased again. The authors interpret this as the model becoming confident that badly distorted images no longer matched their captions. The blur findings were assessed visually from distributions, without a formal hypothesis test.

Other distortions, including noise, compression, occlusion and warping, produced modest increases in mean interval width. Contrast changes left widths essentially unchanged.

Artwork datasets also produced wider intervals than natural-photograph datasets in both short- and long-caption comparisons. This is consistent with uncertainty about unfamiliar imagery, but artwork can also be intrinsically ambiguous. The experiment cannot separate those explanations.

A supplement to human judgment

The experiments cover English captions only, leaving multilingual reliability unknown. Benchmark datasets may also miss rare concepts, styles and real-world conditions.

The preprint therefore supports a qualified conclusion: BEiTScore++ adds informative uncertainty signals without substantially changing scoring performance, but those signals are not universal assurances of accuracy. The authors explicitly position automated evaluation as a complement to human judgment, not a replacement.