Animal ID Benchmarks Can Get the Match Right for Different Reasons

Identifying individual animals in photographs supports research on populations, survival and movement. Automated systems commonly approach this as retrieval: given a query photograph, find another image of the same individual in a gallery. But a successful match leaves an important question unanswered. Was the animal’s appearance sufficient, or could surrounding scenery and other identity-correlated cues also support identification?

In a preprint submitted on November 4, 2026, Ankit K. Upadhyay and colleagues introduce Counterfactual Foreground Retrieval, or CFR. Their proposal does not replace ordinary retrieval accuracy. It adds an evidence profile showing how performance changes when different visual information is removed.

Keeping the task fixed, changing the evidence

The protocol evaluates matched versions of each image while keeping identity labels, valid queries, gallery structure and retrieval scoring unchanged. The full-image view provides the standard benchmark result. A foreground-only view retains the animal—or the annotated body part—and replaces surrounding pixels with a constant neutral fill. A texture-removed shortcut view replaces the animal pixels while retaining the surrounding scene.

The researchers also evaluate a silhouette-only view. Its interpretation depends on the dataset’s intended biometric. Outline can be legitimate identifying evidence for whale flukes or dolphin fins, whereas a benchmark targeting coat markings may treat outline as a shortcut. The supplementary material identifies ambiguous cases and specifies reporting under both definitions.

Importantly, the texture-removed view is not pure background: the filled region still reveals a negative silhouette. Its retrieval score therefore measures recoverable information from scene context together with that animal-shaped region.

Two complementary measures

Retrieval@1 measures whether the highest-ranked gallery image shows the correct individual. Because random success depends on gallery size and the number of matching images, the authors calculate a chance baseline for each retrieval task.

The CFR score takes the smaller of full-image and foreground-only performance after subtracting chance, clipping negative differences to zero and normalizing by the available headroom. High CFR consequently requires both strong ordinary retrieval and strong retrieval when only the foreground remains. It does not reward a model solely for doing well on masked images.

The Shortcut Signal Indicator, or SSI, answers a separate question: how much above-chance retrieval signal is recoverable from shortcut-only inputs relative to the full image? The ratio is capped at one. Where silhouette counts as a shortcut, the calculation uses whichever shortcut view has the stronger signal.

SSI is not the percentage of original decisions caused by background information. High values indicate recoverable identity information under the intervention, not a direct measurement of the model’s internal reasoning or proof of failure.

Similar accuracy can conceal different profiles

The reported results illustrate why both measures matter. On StripeSpotter, MegaDescriptor achieved 99.9% full-image Retrieval@1 and 99.9% foreground-only retrieval, with an SSI of 1.2%. That combination supports a relatively straightforward interpretation: strong matching survives foreground isolation, while shortcut views recover little additional identity signal above chance.

OpenCows2020 presents a different profile. MegaDescriptor reached 100% full-image retrieval and 99.0% foreground-only retrieval, but its SSI was 69.9%. The animal foreground supported identification, yet shortcut-only inputs were also predictive. The authors describe such benchmarks as overdetermined rather than treating the model as simply ignoring the animal.

On NDD20, MegaDescriptor’s full-image retrieval was 85.2%, compared with 35.6% foreground-only and 61.4% shortcut-only retrieval. Its CFR was 34.5% and SSI 71.5%. Under this protocol, the strong original score therefore offers much weaker evidence of foreground-supported identification.

A diagnostic with practical limits

To enable these comparisons, the authors report releasing 457,238 accepted segmentation masks across 34 datasets, covering 94.3% of eligible annotations. A manual audit of 400 images from eight datasets found median mask overlap of 98.2%, measured by intersection over union. Annotations without accepted masks were excluded from the main CFR evaluation, so the results describe that accepted subset.

The team also tested training MegaDescriptor with mixtures of full images and foreground-only inputs. Foreground mixing generally reduced shortcut recoverability, particularly at 50% and 75% exposure, but did not resolve every problematic profile. Reduced SSI sometimes occurred without strong foreground retrieval.

The preprint explicitly frames CFR as a diagnostic, not a causal explanation. Foreground-only retrieval is not guaranteed to be leakage-free, and unfamiliar masked inputs can affect performance. Dataset inspection, split analysis and feature-level interpretation remain necessary to establish where predictive cues originate. The contribution is a more specific evaluation question: not just whether identification succeeds, but which controlled views still make it possible.