A new way to keep multimodal AI’s interactions in check

Why weigh interactions differently?

A remark can sound serious on paper but become humorous through a speaker’s delivery or expression. Systems that detect humor from video need to combine language, audio and visual information. But combining more signals is not automatically better: a model must also decide how much to rely on patterns that appear only when several modalities interact.

Jiahao Qin and Yiwen Wang introduce Möbius Interaction Filtering, or MoIF, to address that problem. It preserves predictions based on one, two or all three modalities, while making the final decision respond less strongly to interactions involving more modalities. The paper is an arXiv preprint, so its claims and comparisons should be read as reported results rather than independently established conclusions.

How the method works

MoIF builds predictions for each nonempty combination of text, audio and vision: three single-modality predictions, three pairwise predictions and one using all three. Separate temporal encoders process the streams. For each pair, reciprocal attention lets each modality use information from the other. The model trains these subset predictions alongside the final combined prediction.

The authors use a mathematical tool called the Möbius transform to describe increments in the prediction associated with different interaction orders. In everyday terms, it distinguishes evidence from a single stream, the additional contribution associated with combining two streams, and the contribution associated with all three together.

For three modalities, the final prediction weights these increments by 4/7, 2/7 and 1/7, respectively. This does not discard the most complex interaction: the model still constructs it, but gives its increment a smaller contribution to the decision. The weighting is fixed by the design rather than learned as a separate filter.

What the experiments found

The main task was humor detection on UR-FUNNY. Across five training runs, MoIF achieved 70.6% accuracy and 70.6% weighted F1. The paper reports that these scores exceeded those of 13 reproduced comparison methods on those metrics. MoIF did not lead on every measure: one comparison system had higher correlation with the continuous prediction target. Different metrics capture different aspects of performance.

The authors also trained and evaluated the approach separately on CMU-MOSI and CMU-MOSEI, datasets used for sentiment prediction. Results were strong on several measures, but the leading model varied by metric. On MOSEI, for example, another method had higher binary accuracy, while its weighted F1 tied MoIF’s. These tests show the method was applied beyond humor detection, but do not establish that it will work equally well in other settings.

To investigate why the filtering might help, the researchers added controlled Gaussian perturbations to encoded features while keeping model parameters fixed. On UR-FUNNY, they found that the ratio of label-aligned signal to perturbation variability declined from single-modality to pairwise to three-way interaction components. Under the tested perturbations, the filtered output’s signal-to-noise ratio was 1.62 to 2.79 times that of the co-trained three-modality prediction. It retained approximately all of its own clean, class-aligned signal. Across five training runs, the authors also report lower prediction variability for MoIF than for the three-modality predictor on all three datasets.

What remains uncertain

These findings concern a particular model and evaluation setup; they do not show that higher-order interactions are generally unreliable. The noise study used artificial changes to internal features, not a broad range of real-world recording problems, and was conducted on UR-FUNNY. The sentiment results do not establish that the same perturbation mechanism applies to those datasets. The authors also note that exhaustively constructing every modality subset becomes costly as the number of modalities grows.

The filter response is fixed, so it may not be optimal for every dataset or task. Results average five training seeds, and comparisons depend on the reproduced implementations and evaluation choices described in the paper. The authors say code is available, which could support further checks. As a preprint, the work is a report of the authors’ findings; additional evaluations and independent checks would help clarify how broadly the approach is useful.