When grading rules leave gaps, AI can quietly fill them

Testing the words, not the clinicians

In a preprint submitted to arXiv on November 4, 2026, Elisa Rovai and Fabio Rovai analyse four texts used to describe diabetic retinopathy grading. Their question is narrow: what categories does the wording actually support, before a reader supplies missing assumptions or resolves contradictions?

The sources are England’s December 2025 screening definitions, the international scale as tabulated by the International Council of Ophthalmology in 2017, its original 2003 publication, and part of a 2015 Italian guideline. The authors explicitly distinguish their analysis of these documents from an assessment of medicine, clinicians or patient outcomes.

Making ambiguity explicit

The researchers translated the texts into rules over 24 variables describing findings such as small vessel abnormalities, haemorrhages and deposits near the centre of vision. Instead of choosing one interpretation of ambiguous wording, they represented alternative readings explicitly. These included different meanings of “isolated” and different ways to resolve unstated rule precedence.

They report exhaustively checking 1,425,807,360 finding profiles. An independent logical solver re-derived the formal claims, while an ontology reasoner checked selected cases. A result described as holding under every reading therefore survives all interpretations included in their formalisation—not every interpretation anyone could conceivably propose.

Missing and conflicting categories

Each of the three texts that assigns follow-up actions contains a case left without a category under every included reading, the authors report. The 2003 scale, which does not provide a referral table, has gaps under some readings but none that survives all of them.

One English example is an eye whose only finding is a hard exudate, a lipid deposit, outside the macula. The wording excludes it from the no-retinopathy category, but requires other findings before an exudate qualifies for the background-retinopathy category. No higher category applies.

The authors also identify overlapping rules. In the Italian table, a single microaneurysm—a small bulge in a capillary—matches both a mild category prescribing annual screening and a moderate category prescribing follow-up in six to twelve months.

The two international-scale texts are not interchangeable in the analysis. Read literally, the 2017 proliferative category requires conditions that exclude the proliferative signs it also demands, making that category empty. Its macular rules also leave a gap for hard exudates without retinal thickening that the 2003 wording covers. The authors find that a simple grade-to-grade conversion cannot fully reconcile the versions. England and Italy also rank some pairs of finding profiles in opposite orders, rather than merely placing thresholds differently.

What photographs cannot establish

The researchers then examined 927 annotated photographs from IDRiD, DDR and DIARETDB1. Their initial analysis treated unrecorded findings as absent, including retinal thickening. That assumption is consequential: a single colour photograph cannot establish whether the retina is thickened.

Under that assumption, England and the 2017 international text disagreed on referral for 153 of 570 DDR images on which both gave a definite decision: 26.8%. This is not a disagreement rate for a screening population. The datasets contain hospital images selected for annotation and are enriched for disease.

When thickening was left unknown, every previously identified disagreement involving England became undecidable. This does not establish that the texts agree on those eyes; it means the photograph does not supply enough information to settle the comparison.

Labels shape what models learn

A pre-registered classifier experiment used the 310 images receiving definite referral labels from all three action-setting texts. Models trained on different texts generally learned their respective decisions. All four statistically testable comparisons, which involved England, supported the predicted reduction in agreement when evaluation switched to another text’s labels. Comparisons between the international and Italian labels had too few disagreements for the registered test.

The authors call the overall result partially supported. Crucially, the English label differences depended on assuming no retinal thickening, so the models reproduced labels carrying information the photographs could not establish.

Language models supplied silent repairs

A separate pre-registered study tested language models on grading problems, including 13 gap items and 27 overlap items. Qwen3.8-Flash-Next and Claude Haiku 4.5 without deliberation often returned one category even when the text licensed none or several, despite instructions to read literally. Claude Sonnet 5.5 and Claude Opus 5.5 reported defects more often.

Switching deliberation on reduced repairs within the two models tested both ways, but did not eliminate them; statistical support varied by model and instruction.

The study’s limits include small defect-item samples, one response per Claude condition, incomplete annotations and a shared rule transcription that could propagate errors across checking engines. The practical implication is to document the exact grading text and assumptions behind AI labels, rather than treating a scale’s name as a complete specification.