When does a second AI opinion help retinal screening?

A screening system that can defer

In a November 2026 preprint, Sohaib Sulman and colleagues describe RetinaGuard, a framework for deciding whether to accept an automated retinal screening result or refer the photograph for clinician review. The study evaluates a computational triage policy, not a deployed clinical service.

The researchers define an automated decision as unsafe if its normal-versus-abnormal screening answer is wrong, or if it misses at least one disease present in the image. False-positive disease findings do not count as unsafe under this definition, because the authors assume they lead to review. This is a specific operational definition, not a measure of every possible clinical harm.

Development used the public RFMiD dataset. The researchers trained models on 1,536 photographs, reserved 384 for statistical certification, used 640 validation images for model selection and fitting other components, and evaluated on 640 test images. The task covered 25 labels: one normal-versus-abnormal screening label and 24 disease labels.

What the agents actually do

Here, an “agent” is a software component controlling evidence gathering or routing—not necessarily a conversational AI. An image-quality component first checks photographic properties and refers acquisition outliers before diagnosis.

A RETFound model then produces screening and disease predictions. Twenty stochastic passes through the model provide uncertainty measurements, including how often its predicted label set changes. A learned failure predictor combines these signals to estimate the chance that the decision is unsafe.

An independently trained ConvNeXt-B model can supply a second opinion. Its answer does not replace RETFound’s prediction; disagreement becomes additional evidence about whether review is needed. Another component can calculate explanation-map features. Additional tools run only when their predicted value for the decision outweighs their computational cost.

Verification helps unevenly

The researchers repeated the analysis with two RETFound models trained using different fine-tuning schedules. Their macro-AUROC scores were 0.896 and 0.925. These measure ranking performance averaged across all 25 labels—the screening label and 24 disease labels—not diseases alone.

With the weaker primary model, verifier disagreement supplied information beyond uncertainty. At a referral budget of 20%, RetinaGuard captured 119 of 280 unsafe decisions, compared with 88 using lowest-confidence referral. Because both methods referred the same number of images, this comparison tests which policy found more errors without increasing review volume.

With the stronger primary model, the verifier added little beyond uncertainty. At the same 20% referral budget, RetinaGuard captured 82 of 229 unsafe decisions, compared with 68 for lowest-confidence referral. Across the tested budgets, none of its comparisons against calibrated-uncertainty referral remained statistically significant after correction for multiple comparisons. The stronger model’s uncertainty already contained much of the useful warning signal.

What the safety target means

RetinaGuard uses held-out data and conformal risk control to set its acceptance threshold. Crucially, its main guarantee concerns unsafe acceptances as a fraction of all cases, not the error fraction among accepted cases. It holds in expectation under an exchangeability assumption: calibration and future cases must be statistically interchangeable.

At a nominal 5% target, RetinaGuard accepted 37.0% of test images with the weaker model and 37.7% with the stronger one. The realised marginal unsafe rates were 5.6% and 5.5%, respectively. These single-test-set results illustrate why a nominal target is not a promise that every sample stays below it.

The quality checks helped under synthetic image degradation, but did not restore that target reliably. With the stronger model, they reduced the worst marginal unsafe rate across tested corruptions from 53.1% to 28.0%—an improvement, but still far above 5%.

Costs, explanations and limits

Selective tool use reduced estimated computation by about 30%. The policy never invoked the explanation component. These savings came from replaying routing decisions using stored tool outputs and measured per-tool latencies, rather than timing an end-to-end deployment.

Separate tests on 81 images with lesion masks also cautioned against equating stable visual explanations with meaningful ones: RETFound attention-rollout maps were stable under perturbation but did not localise lesions well.

External evaluation exposed the largest safety limitation. On ODIR-5K, screening AUROC fell from 0.974 to 0.663. Despite referring 87.4% of images, the policy’s marginal unsafe rate reached 6.8%, exceeding the 5% target. The guarantee did not transfer to that population.

The evidence remains limited by a 640-image main test set, keyword-derived external labels, synthetic quality degradations and only two primary-model training settings. The preprint supports a narrower lesson than “more agents are safer”: extra evidence helps when it reveals otherwise hidden errors, and statistical safeguards remain dependent on the conditions under which they were calibrated.