StakeBench asks whether AI can read what market traders have at stake

A prediction-market comment can sound confident without revealing whether its author has actually taken a position. Conversely, a trader’s wording may make more sense when read alongside what they stand to gain or lose. StakeBench, a new evaluation framework, asks whether language models can make that connection—and whether they can go beyond reading a position to anticipate what happens next.

The study, by Yunhua Pei and colleagues, links 560,876 comments from 2,261 resolved markets on Polymarket and Manifold to public records of users’ positions, trades and market odds. The paper describes the work as an accepted author manuscript for NeurIPS 2026. Instead of asking human annotators to label comments as bullish, bearish or neutral, the researchers use recorded market behavior to construct targets. The idea is to evaluate what speakers have observably committed to, rather than how their words strike an outside reader.

That distinction matters. A person’s position is evidence of a choice, but it is not a direct window into their private belief: people can trade for reasons other than confidence, and they may express themselves strategically. The authors therefore frame these records as behavioral proxies, not measurements of hidden mental states.

Four steps from words to market behavior

StakeBench tests four increasingly demanding tasks. First, can a model tell whether a commenter holds a position in the market? Second, can it identify which side that position supports—YES or NO? Third, can it predict whether the trader will later flip, increase, decrease or hold the position? Finally, can it use groups of comments to project whether market odds will rise or fall?

For the action task, the target is the first qualifying position change within 48 hours of a comment. For the odds task, the researchers compare models with a baseline based on the commenters’ revealed positions. This asks whether language adds useful direction information beyond what the positions already show—not simply whether comments and later odds move together.

The team evaluated 15 language models across 18 topic-and-platform settings. Scores were averaged across those settings, so the biggest discussion pools did not dominate the results. They also introduced metrics intended to account for position size and compare model predictions with reference signals from observed behavior.

A steep drop from detecting a position to predicting an action

Models showed some ability to identify a commenter’s revealed side, but performance was modest: Directed Accuracy ranged from 0.506 to 0.599. This score counts directional predictions when a model does not abstain, so it should not be read as accuracy across every example. The result also does not mean models reliably uncover private beliefs; it concerns whether their predictions match the recorded position side.

The next task exposed a different weakness. Ten of the 15 models relied heavily on just one or two of the four possible action labels. In other words, they often failed to distinguish a trader who would hold from one who would change their exposure. At the group level, no model consistently beat the position-based baseline for projecting the direction of market odds.

The paper reports that model size did not correlate with performance, and finance-domain tuning did not improve revealed-side identification. Results also varied by platform, especially for later-action prediction. The authors suggest that differences in platform incentives may matter, but the platforms are not interchangeable: one uses real money, while Manifold uses virtual currency, and their position records differ in coverage.

What the benchmark cannot establish

StakeBench does not show that comments cause odds to move. Nor does its odds result establish that comments contain no signal: the comparison is specifically about whether models add direction information beyond a position-weighted baseline. The paper notes that market efficiency and model limitations could both help explain the result, and its additional text-probe analysis does not settle that question.

The dataset also has uneven coverage. Position records cover a smaller share of Polymarket commenters than Manifold commenters, and fewer Polymarket records receive labels for later trading actions. The authors say they cannot determine whether the unreconstructed Polymarket records differ systematically from the retained ones. The benchmark uses English-language comments, fixed prompts and a selected set of resolved markets, so its results do not establish how models perform across other languages, market types or prompting strategies. Public comments may also have appeared in models’ training data.

The paper presents StakeBench as an evaluation tool, not a new model or a definitive ranking of language models. Its main contribution is a way to test several distinct questions—position detection, side identification, action prediction and odds projection—against records of observable behavior. The authors provide the evaluation code and dataset under a CC-BY 4.0 license. As with any preprint or accepted manuscript, readers should consult the linked source for the full methods and qualifications; the paper’s results are evidence about this benchmark and its protocol, not a general verdict on AI or financial markets.