Can one medical-image detector learn across X-rays and mammograms?

A medical-imaging model can only learn from the labels it is given—but labels are costly, and datasets do not always use the same vocabulary. One set of chest X-rays might mark lung anatomy, another might identify disease, and mammograms have their own findings and annotation practices. Can a detector learn from these mismatched sources without forgetting what it already knows?

Sheethal Bhat and colleagues describe two approaches to that problem in a study of lesion detection. Their aim is a model that can locate findings across chest X-rays (CXRs) and mammography, rather than one trained for a single dataset. The work is presented as a foundation-model approach, but its evidence comes from a defined collection of datasets and tasks—not from testing every kind of medical image or clinical setting.

Making different labels work together

The researchers used four datasets: three involving CXRs and one involving mammograms. The CXR data included labels for anatomy as well as abnormalities; the mammography data included findings such as masses. Some datasets had bounding boxes showing where a finding was, while others provided different kinds of labels.

The first method, called the Anatomy-Abnormality Foundation Model (AAFM), addressed gaps in the CXR labels. Separate models trained on individual datasets acted as “teachers,” generating proposed labels for categories missing from other datasets. The researchers retained selected predictions as weak labels and trained a final model using those predictions alongside expert-provided annotations. This is a way of making more of the available images usable, but the generated labels are not equivalent to labels verified by radiologists.

The second approach, the EM-DETR Foundation Model (EM-FM), builds on a detector that uses visual examples of a finding as guidance. During training, the model learns class-specific visual representations from annotated regions, alongside text labels. At inference—the stage when it analyzes an image—the annotated examples are not supplied; the stored representations guide its search. The researchers trained this model using CXR and mammography data. They compared its results with a baseline detector and with the earlier, single-dataset EM-DETR approach.

Better localization in the reported tests

For four CXR findings, AAFM reached an average image-level AUC of 0.972, compared with 0.962 for the reference product solution and 0.959 for the baseline. AUC measures how well a model distinguishes images with and without a finding; it does not show whether the model precisely locates it.

For localization, the study reports mean average precision at 50% intersection-over-union (mAP50), a measure that rewards predicted boxes overlapping the annotated regions. EM-FM’s average CXR mAP50 was 0.567, compared with 0.527 for EM-DETR and 0.369 for the baseline. For mammography mass detection, EM-FM scored 0.721 versus 0.699 for EM-DETR. The paper reports statistically significant improvements for several comparisons, but not for every finding or measure.

The results were not uniformly best across tasks. Atelectasis detection showed little improvement, and the authors point to ambiguity in how that finding is labeled and how it can resemble other conditions. At a fixed rate of false positives, EM-FM’s average sensitivity across the four CXR findings was 0.927, below the reference product solution’s 0.954. A model can therefore perform well on one summary metric without leading on another practical measure.

Promising, but not a clinical test

The study suggests that combining datasets carefully—and using example-based guidance—can improve detection localization on these evaluations. It also shows why simply pooling data may not suffice: the authors report that naive combinations could sharply reduce performance for some findings. Their proposed training strategy is intended to mitigate that problem, not to prove it has been solved for medical imaging generally.

Important limits remain. The experiments cover selected findings and datasets, including a mammography dataset drawn from a Vietnamese population with high breast density. The authors say future work should add findings and datasets to test robustness and generalizability. The results are retrospective model evaluations, not evidence that using EM-FM improves patient outcomes or that it is ready to guide care.

This article is based on the arXiv version identified as 2611.14518v1. A preprint is a research manuscript, and the supplied source does not establish that this particular study has undergone peer review. Its reported comparisons should be read in that context; further independent evaluation would be needed before drawing conclusions about performance in routine clinical practice.