Medical image AI faces a cross-site test

Medical image segmentation asks software to outline structures or lesions in an image, pixel by pixel. Those outlines can matter for measuring disease or supporting clinical review. But a model may learn features tied to its training data: the equipment, image appearance or patient mix. That makes performance on a new hospital’s data a practical test, not just a technical detail.

The authors say that many comparisons between foundation models and task-specific systems have tested only data resembling the training set, and have relied on a single task-specific baseline whose configuration may not be well justified. Their study instead compares an adapted foundation model with models trained from scratch, including tests on independent datasets from other institutions.

Comparing adaptation strategies

The foundation model was DINOv2, a model pretrained on natural images. The researchers adapted it for medical segmentation using strategies ranging from changing only bias terms to fine-tuning the full model. They compared these approaches with U-Nets and ResU-Nets trained from scratch. The experiments covered colonoscopy, breast ultrasound and dermoscopy—images of the skin—and evaluated both held-out data from the training source and external datasets.

The abstract does not provide the number of images or institutions, the specific performance metrics, or detailed results for each task. It therefore supports a broad account of the comparison, but not a close assessment of how large or consistent the reported differences were within each setting.

A trade-off between familiar and new data

The authors report that the adapted foundation model generalised better on every external dataset they tested. On data from the training distribution, however, task-specific models remained competitive when training data were abundant. They also delineated boundaries more precisely, while the foundation model more often missed small lesions.

That contrast matters: a model’s advantage depends on what it is being asked to do and where its images come from. Stronger performance across the tested external datasets does not establish that the foundation model is best for every clinic or every lesion. And sharper boundaries on familiar data do not, by themselves, show which approach would be more useful in clinical practice. The abstract does not report clinical outcomes or describe a real-world deployment.

The authors also say that only a small fraction of the foundation model needed adaptation, and that the capacity required was determined by the imaging domain rather than by the adaptation method. In plain terms, the amount of model adjustment that was useful appeared to depend on the kind of images—such as colonoscopy versus skin imaging—not simply on which tuning strategy was used. The abstract does not give the exact fractions or measurements behind this conclusion.

Data checks matter, too

A separate finding concerns evaluation data: the authors report that a widely used polyp test set duplicated images from another benchmark. If test images overlap across benchmarks, comparisons using those benchmarks may not represent fully independent tests. The abstract does not name the datasets or explain the extent of the duplication, so its consequences cannot be judged from this evidence alone.

The work is an arXiv preprint, and the material available for this article is its abstract rather than the full paper. That limits what can be checked: details of dataset selection, training protocols, statistical uncertainty and the duplicate-image finding are not available here. The results should therefore be read as the authors’ reported findings, not as independently verified evidence or a clinical recommendation.

Their cautious takeaway is that adapting a foundation model may be worthwhile when a segmentation system must work across sites, while the amount of adaptation should be chosen for the target imaging domain. The reported trade-offs also argue against assuming that one model type will excel on every dataset or every segmentation detail.