Why a social-media post may not reveal its author’s stance

Researchers often ask annotators to label a social-media post: does it support a cause, oppose it, or express a positive or negative feeling about a topic? But if the intended label describes the author’s perspective, a single post may not provide enough evidence. That is the problem explored in a preprint by Yunpeng Xiao, Kyrie Zhixuan Zhou, Youpeng Zhao and Kai Shu.

The authors call such tasks “individual-level” natural language understanding. Their point is that a person’s stance is not necessarily the same as what a sentence seems to mean on its own. A post may be ambiguous, refer to unfamiliar cultural or political context, or include irony or a quotation. When annotators lack relevant information, they may assign a label that does not match the author’s perspective.

The researchers selected three existing datasets: a SemEval tweet dataset for stance, another SemEval dataset for topic-based sentiment, and HateXplain for hate-speech classification. For the stance and sentiment samples, they collected additional posts by the same users, generally from a bounded period and related to the same target or topic. Three annotators—the paper’s authors—then reassessed the labels using this added context. Their guidelines treated multiple posts as evidence about a user’s perspective and set out how to handle unclear or conflicting evidence. For hate-speech samples, user context served as supporting information alongside social definitions of hate and offensive speech.

This approach depends on an important assumption: that a person’s stance or sentiment toward a topic is stable enough over a limited period for other posts to help interpret a particular post. The authors focused on targets they considered more likely to reflect durable views, and excluded politician-centered targets from their stance re-annotation because attitudes may shift with events. They also report that some original posts and user histories were unavailable.

Under their re-annotation, the researchers report label-error rates of 31.7% for the selected stance samples and 23.3% for topic-based sentiment. For HateXplain, they examined inconsistent test-set posts; of 350 such posts, only 76 were still accessible and could be included. The authors report a 48.7% error rate for those accessible posts. These are estimates under the study’s annotation approach, not rates established for every item in the original datasets or for social-media benchmarks generally. Annotator agreement was lower for hate-speech labels than for the other tasks, consistent with the difficulty of separating hate speech, offensive language and normal speech.

The team also tested GPT-4o, Llama 3-70B and Phi-4 against the revised labels. On stance, all three models scored higher against the revised labels than against the original labels in the reported results. On sentiment, all three also scored higher against revised labels. The paper’s table gives the specific scores: GPT-4o rose from 64.77% to 92.46% on stance and from 78.33% to 89.17% on sentiment; Llama 3-70B rose from 64.56% to 87.58% and from 78.33% to 92.50%, respectively; Phi-4 rose from 61.51% to 79.02% on stance and from 76.67% to 92.50% on sentiment. These comparisons show performance against two sets of labels, not independent proof that either set perfectly captures a person’s perspective.

In an ablation experiment, model accuracy on all three tasks was higher with multiple posts than with a single post. In a separate stance experiment, performance continued to improve as more posts were provided, though the gains became smaller; the authors suggest three posts as a practical balance in their setting. The results support the idea that context can matter, but do not show that models can reliably recover people’s private beliefs.

There are substantial uncertainties. The revised labels are judgments by three annotators, who discussed disagreements until they reached consensus—not objective records of inner states. People can change their minds, post inconsistently, or express themselves differently across situations. The hate-speech findings are especially hard to generalize: they concern only the 76 accessible posts, and the categories themselves have contested boundaries. The selected samples and missing posts also limit what can be concluded about the original datasets as a whole.

This work is an arXiv preprint, so its claims should be treated as provisional, not as settled findings established through peer review. Its practical message for dataset builders is narrower: when labels are meant to capture a person’s perspective, annotation instructions and evaluation data should clarify whether that person’s wider context is relevant. The study does not establish that all text-only labeling is flawed. It highlights why, for some questions, a sentence alone may not support a confident judgment.