Can multimodal AI learn to use its inputs together?

A sarcastic remark may be easier to interpret when an AI system considers facial expression and vocal intonation together. In multimodal machine learning, useful predictive information that arises only from combining inputs is often called synergy. Yet giving a model several kinds of data does not ensure it will use them jointly. One modality can dominate training while others contribute less.

A new preprint proposes the Multimodal Synergistic Competition Regularizer, or MSCR, to tackle both issues: discourage excessive reliance on a single modality and encourage the fused predictor to outperform a specific reference built from unimodal predictions.

Balancing reliance and joint predictions

The researchers tested MSCR on five benchmarks: CREMA-D for emotion recognition, AVE for audio-visual event recognition, UCF101 for action recognition, and CMU-MOSI and CMU-MOSEI for sentiment analysis. The experiments cover two- and three-modality setups. The main comparison is with the Multimodal Competition Regularizer (MCR), which is designed to balance modality contributions during training.

MSCR adds two objectives. Its anti-dominance objective measures how much a fused prediction changes when one modality’s internal representations are shuffled across examples. It penalizes large changes, treating them as a proxy for reliance on that modality. This sensitivity is not a direct measure of how much useful information a modality contains. Shuffling may also disrupt relationships between modalities.

Its synergy objective compares the fused prediction with an additive reference: the average of the individual modalities’ predictions. The objective encourages the fused predictor to achieve a lower task loss than that reference by a specified margin. This is a particular comparison, not a general test of whether fusion beats every possible combination of unimodal evidence.

An adaptive gate scales both objectives using the smallest unimodal loss for each example. In effect, it applies more regularization when this proficiency measure is above a chosen threshold, and less when it is below. The gate is therefore tied to a defined loss-based proxy, not a universal assessment of difficulty.

MSCR’s own gradients update the fusion network, while the full training objective also includes task losses and MCR regularization, which update parameters along their designated paths, including the modality encoders. The authors follow MCR’s evaluation protocol and report means across three folds or three random seeds, depending on the benchmark. They reproduced MCR results; results for other baseline methods in the main comparison were taken from the earlier MCR study.

Better scores and more conditional recoveries

In the main results, MSCR scores higher than MCR in each reported setting. The authors also assess conditional recovery: how often the fused model is correct on test examples where all the corresponding unimodal predictors are wrong. This describes whether fusion succeeds on cases that individual predictors miss; it is a behavioral indicator of synergistic capability, not a direct measurement of information content.

In the CMU-MOSI video-and-text setting, for example, the reported conditional recovery rate is 34.35% for MSCR and 4.85% for MCR. MSCR’s rate is higher in every setting in the paper’s comparison. The authors also use Global InterSHAP, an output-attribution method that estimates the relative contribution of cross-modal interactions. In three reported settings, MSCR has higher interaction scores than MCR. These measures offer complementary evidence: one counts successful predictions on unimodal failures, while the other attributes parts of model output to interactions.

In a further test, the researchers replaced the default fusion layer with a Transformer-based module. MSCR still had higher conditional recovery than MCR in all four reported settings. In selected ablation experiments, either MSCR objective alone improved on the MCR baseline, and using both produced the best results in those settings.

Together, these findings support the authors’ claim that MSCR can improve performance on the tested benchmarks and increase measures associated with cross-modal collaboration. They do not establish that a particular internal representation or a specific new shared pattern has emerged.

What the evidence cannot settle

The paper acknowledges that averaging unimodal predictions is a tractable but limited reference: it may not capture every form of interaction. Beating it is not an exact measure of synergy. Likewise, reducing sensitivity to shuffled inputs is a design motivation for leaving room for joint cues, not a guarantee that those cues will form. Conditional recovery and Global InterSHAP are useful proxies, but neither is a definitive information-theoretic measurement of what the model learned.

The findings are also limited to five benchmarks and the models and protocols tested. Main results are averaged over three folds or seeds, but the supplementary conditional-recovery matrices use one run per setting; Global InterSHAP scores are averaged across folds or seeds. Results from other baselines were not all reproduced by the authors, which makes the comparisons less uniform than a fresh, controlled re-evaluation of every method would be.

This is an arXiv preprint, which does not by itself establish peer review. The authors provide code, but independent replication and testing on a wider range of tasks would help show how robust the results are. For now, MSCR is a promising method for encouraging balanced use of modalities and better fused predictions, with synergy assessed through practical proxies rather than proven directly.