Combining biopsy images and clinical data: no single AI method wins across bowel disease outcomes

Biopsy images and clinical measurements describe different aspects of inflammatory bowel disease. Histology captures microscopic inflammation, while blood tests, stool measurements and other clinical information provide complementary context. A new study asks how best to combine those sources when predicting later remission—and finds no universally strongest approach.

Ankana Banerjee and colleagues compared 15 fusion methods across six prediction scenarios. Their paper, posted to arXiv on November 3, 2026, is a preprint listed as submitted to the 2027 IEEE International Symposium on Biomedical Imaging. That submission does not establish conference acceptance or completed peer review.

Three outcomes, two starting points

The researchers used the prospective, single-center SUSI-IBD cohort of newly diagnosed patients. They predicted remission at an 11-month visit using information from either diagnosis or the three-month follow-up. Each input visit was evaluated separately, rather than combining both visits into a longitudinal prediction.

The three outcomes reflected different kinds of disease control: histological remission assessed from tissue, endoscopic remission assessed through examination of the bowel, and biochemical remission defined by C-reactive protein and faecal calprotectin thresholds. The paper frames these outcomes around STRIDE-II, while distinguishing histological remission as a measure of deeper healing alongside its treatment targets.

Depending on outcome availability, the analyses included 250–265 patients for diagnosis-time inputs and 241–258 for three-month inputs. Patients were split between datasets at the patient level to prevent leakage between training and evaluation.

Clinical inputs included age, body mass index, laboratory measurements, clinical and endoscopic scores, complication indicators, sex, medication and disease subtype. The Nancy Histologic Index was excluded from this branch so that histological information entered through the images rather than as a clinical score.

Comparing ways to combine information

The image pipeline divided digitized biopsy slides into tissue patches and encoded them using the frozen Virchow2 foundation model. An attention-based model trained for histological grading produced slide representations, which were then frozen for the fusion experiments. Trainable components subsequently combined information across biopsy locations and with the clinical representation.

The fusion methods ranged from straightforward concatenation—placing image and clinical representations side by side—to gating, modulation, bilinear interactions and attention. Other methods combined the predictions of separate image and clinical classifiers instead of merging their internal representations.

The framework standardized much of the surrounding prediction machinery. Early and intermediate fusion methods used a common classifier design, although its input width differed for some methods. Decision-level approaches instead combined separate modality predictions. Image-only and clinical-only models provided the comparison baselines.

Better point estimates, uncertain gains

The best fusion model in each scenario achieved a macro-F1 score between 0.57 and 0.75. Macro-F1 averages the F1 scores for the prediction classes; it is not the percentage of patients classified correctly.

By point estimate, fusion outperformed both single-modality baselines in four scenarios. With diagnosis-time inputs, it was comparable to the image-only model for biochemical remission and matched the clinical-only model for endoscopic remission.

The winning approach changed with the task. Compact bilinear pooling led for biochemical remission using diagnosis-time inputs, whereas early feature-wise modulation led for histological remission using three-month inputs. Across scenarios, method rankings showed little consistency: the reported mean Spearman correlation was −0.08. Every scenario also had at least one alternative method within 5% of the top macro-F1.

These results need substantial qualification. The displayed confidence intervals for gains over the stronger single-modality baseline all included zero. Each model was trained only once, so bootstrap intervals represented test-patient sampling uncertainty, not variation between training runs. The manuscript also gives inconsistent test-set sizes: 26–31 patients in the experimental description versus 33–35 in Figure 3.

Accuracy does not reveal image dependence

The team separately tested image reliance by replacing slide representations with zeros while leaving trained models unchanged. The resulting drop in balanced accuracy measured sensitivity to removing that input. Models performing at or below chance were excluded from this analysis.

Reliance tended to be highest for histological remission and lowest for endoscopic remission, but differences between outcomes were not statistically significant. The strongest model in each scenario lost 0.06–0.23 in balanced accuracy without images. Yet 21–50% of evaluated methods, depending on scenario, showed no decline or improved after image removal.

Similar prediction scores therefore did not mean similar use of the two information sources. Nor did the comparison establish a robust universal winner: top methods were identified on the test sets. The authors call for larger cohorts, repeated training, multi-timepoint inputs and external validation. Their central recommendation is to compare several fusion methods across outcomes and report image reliance alongside predictive performance.