When words, voice and faces disagree, an emotion AI changes its approach

What should a computer make of a conversation when words, tone and facial expression seem to tell different stories? Combining all three does not automatically produce a better answer: an unreliable signal can interfere with a useful one.

In an arXiv preprint submitted on November 6, 2026, Jingwen Wang and colleagues propose MiD-ERC, a system that adjusts its processing according to how strongly its text, audio and visual emotion predictions disagree. The task is to assign an emotion category to each conversational utterance, using its signals and dialogue context. It is not a direct measurement of someone's internal feelings.

Measuring disagreement before combining evidence

The system starts with pre-extracted text, acoustic and visual features. It processes each stream separately, tracking conversational history and distinguishing earlier contributions by the same speaker from those by other speakers. Its graph-based context module connects an utterance only to preceding turns within a limited window.

Each stream then predicts probabilities over the same emotion categories. Rather than comparing only their top choices, MiD-ERC compares the full probability distributions using Jensen–Shannon divergence, a measure of how different distributions are. The authors normalize this score to a range from zero to one.

A low score means the three predictors broadly agree; a higher score means their predictions differ more. Crucially, this measures disagreement between models. It does not establish whether a speaker is being sarcastic, concealing emotion, or simply recorded under difficult conditions. The framework deliberately avoids diagnosing those causes separately.

Disagreement also cannot tell the system which stream deserves more trust. A separate estimator therefore learns a reliability score for each stream, using its contextual features, predictions, confidence, uncertainty and the overall divergence. During training, its target is the probability that the corresponding single-stream classifier assigns to the annotated emotion. This is a model-based definition of reliability, not independent verification of a person's feelings.

Three ways disagreement changes the decision

MiD-ERC uses these signals at three stages.

First, a learned gate controls how much information the streams exchange through cross-attention. The authors' analysis finds that greater predictive disagreement tends to reduce this exchange, limiting the injection of competing evidence into each representation.

Second, the system combines the streams using reliability-based weights. Higher divergence makes those weights more even, while an additional training penalty discourages one stream from dominating highly disputed cases. That is a distinctive choice: rather than simply discarding the apparent losers, the system preserves competing information for the final decision.

Third, it changes the instructions given to a large language model. When disagreement is low, the instructions emphasize the combined representation, with individual streams as supporting evidence. When disagreement is high, they ask the model to compare the separate streams against the dialogue context instead of relying only on the combined signal or the most confident predictor.

The main experiments use Qwen2.5-7B-Instruct, with its base parameters frozen and small trainable adapters added. The multimodal modules and adapters learn from ordinary emotion annotations, without extra conflict labels or external retrieval. The selected instructions and feature representations enter a single language-model forward pass.

What the benchmark results show

The researchers evaluate MiD-ERC on IEMOCAP, which contains acted two-person conversations, and MELD, drawn from the television sitcom Friends. Results are averaged over six random seeds.

MiD-ERC achieves weighted F1 scores of 73.38% on IEMOCAP and 70.60% on MELD. F1 balances precision and recall; its weighted version gives categories influence according to their frequency. These scores exceed the paper's reproduced Causal-ERC results by 1.92 and 1.58 percentage points, respectively.

The reported advantage is larger in the high-divergence groups: 2.87 points on IEMOCAP and 3.88 points on MELD. It is not universal: on low-divergence MELD examples, MiD-ERC performs slightly worse. Removing the adaptive relation instructions also reduces overall weighted F1 on both datasets, supporting their contribution within this architecture.

What remains uncertain

These are benchmark findings, not evidence that the system reliably interprets everyday emotions. Acted laboratory exchanges and scripted television dialogue do not establish performance across other populations, languages or recording conditions. Weighted averages also do not guarantee equally good recognition of every emotion.

The comparison table mixes results taken from earlier publications with reproduced results, and the main performance table does not provide uncertainty intervals. The work is an arXiv preprint; the supplied record does not establish peer-review status.

This account uses the supplied PDF text extraction. Images were unavailable, and equations, tables and layout may be damaged. The central finding is therefore a bounded one: explicitly responding to predictive disagreement improved the reported benchmark results, without establishing why the underlying signals disagreed.