When a Virtual Person’s Words and Body Tell Different Stories

Many systems that animate virtual speakers choose gestures and expressions from the words being spoken. That can make a character’s body match or illustrate its sentence, but it leaves little room for irony, tact, or feelings that slip through despite what someone says.

In When Words and Bodies Disagree: Realizing Verbal and Nonverbal Mismatch, Parisa Ghanad Torshizi and Stacy Marsella explore whether a large language model (LLM) can select mismatching behavior when given the broader social situation. Their work is about choosing and evaluating expressive behavior for a virtual human—not establishing how people’s bodies reveal their true feelings in everyday life.

Three ways words and behavior can diverge

The researchers organize mismatches by their social function. Strengthening is intentional: a speaker uses a cue such as a deadpan expression to make sarcasm clearer. Mitigating is also intentional: warm behavior can soften criticism, or a restrained response can play down a compliment. Leaking is unintentional: a feeling someone would rather conceal shows through, as when distress appears alongside a claim that everything is fine.

These categories matter because a mismatch is not necessarily a mistake. Its meaning depends on the relationship, conversation, and situation—not just the current sentence.

Giving the model more than a sentence

The authors created 18 two-person scenarios: three for each of six mismatch cases. Each included a relationship and setting, a short dialogue history, and a final line for the target speaker. The scenarios were authored by the researchers or generated with a language model and then curated.

They tested Claude Sonnet 4.6 under four prompt conditions: the final utterance alone; the utterance plus dialogue; those inputs plus a context description; and, finally, the same information with an explicit label naming the desired mismatch case. Each scenario-condition combination was sampled three times. The model proposed facial actions, gaze, head movement, and prosody, and identified the relationship it intended those cues to convey.

In a separate evaluation, the team turned selected behaviors into videos using the SmartBody virtual-human framework. Participants recruited through Prolific viewed the clips and rated the character’s attitude, emotion, or apparent social function. For the first two studies, researchers compared matching and mismatching versions of selected scenarios; a third study examined ratings of strengthening, mitigating, and leaking.

Context helped—but did not settle every case

As more information was provided, the model selected a mismatch in more scenarios: 4 of 18 with the utterance alone, 12 with dialogue, 16 with dialogue and context, and 18 when explicitly told which case to produce. The context-only condition fell short of the explicit instruction on two scenarios. Mitigated criticism was especially difficult: the model chose a mismatch in just one of its three scenarios with dialogue and context, versus all three when the case was named directly.

In the video studies, nonverbal cues shifted participants’ interpretations even when the spoken line stayed the same. For example, viewers tended to see a positive sentence paired with negative cues as less positive than the same sentence paired with positive cues. In the study of social functions, mismatching clips received higher ratings for their intended function than matching baselines. But the functions were not equally easy to tell apart: leaking was more distinct, while mitigation was difficult to separate from the other categories; strengthening and leakage also overlapped in some judgments.

What the study cannot yet tell us

The results support a focused conclusion: in these authored scenarios, one LLM used dialogue and situational context to select behaviors that viewers often interpreted in the intended direction. They do not establish that the model reliably understands people’s hidden mental states, or that these cues would be read the same way in unscripted interaction.

The evaluation is limited in scope. The human studies used one scenario per mismatch case, a single LLM, one animated character, and English-speaking online participants. The authors caution that the findings may not generalize across model families or cultures. The character system also required manual adjustments to map the model’s suggestions into animation; it could not realize vocal prosody, and the studies did not include gestures or posture. Participants in the third study saw the videos without the surrounding context, so they had to infer meaning from behavior alone.

There is also a practical concern: the same technique that could help train counselors or build expressive virtual agents might be used to make an artificial character appear sincere, sarcastic, or emotionally revealing on demand. The paper reports a preprint manuscript, not evidence of broad real-world effectiveness. Its contribution is a promising design direction—and a reminder that expressive AI needs careful evaluation as well as richer cues.