When an AI’s World Model Runs, Does It Describe the Right World?

A working model can still be the wrong model

A formal world model is a simplified description of who is involved, what they can do, and what may happen next. Planners and safety tools can use that description to explore possible outcomes. Large language models (LLMs) might make it easier to create such models from ordinary text—but a model that executes without errors is not necessarily a faithful account of the situation.

That distinction is the focus of a study by Souhaila Djaffal and colleagues. The authors prompted five LLMs to turn scenarios into partially observable stochastic games, or POSGs. In plain language, a POSG lays out multiple players, possible states and actions, uncertain transitions, and what each player can observe. The researchers’ main question was whether mechanical validity—whether a model is well-formed and behaves consistently—also tells us whether it represents the intended decision problem.

Three kinds of checks

The team assembled 44 scenarios across toy environments, games, open-ended situations, and culturally contextualized cases. They analyzed 410 first-pass model generations across three evaluation tracks: games with authored reference models, situational cases with criteria-derived references, and cultural cases tested with and without a country name in the prompt.

Their evaluation separated three questions. First, mechanical validity: can the model run, and are its states, actions, transitions, and probabilities internally consistent? Second, structural agreement: where a reference exists, how closely does the generated model match its states and transitions? Third, semantic checks: does it include expected features and avoid specified unsupported ones?

For the controlled game track, researchers compared models after aligning differently named players, states, and actions. They limited the comparison to three transitions from the initial state, making it possible to calculate a structural F1 score. That score summarizes how much reference structure a candidate reproduces while accounting for extra structure. It is useful, but it is not a complete test of meaning: the comparison can penalize reasonable differences in granularity, and behavior beyond the three-step horizon is not assessed.

Passing the test is not the same as matching the reference

In the game track, 134 of 135 generated models passed the strict mechanical audit. Yet their mean structural F1 was 0.366, and only 28 of the 134 mechanically valid models exactly matched the authored reference. The authors report that game models tended to add structure, such as extra states or transitions. In one example, a model introduced bid amounts that the scenario explicitly ruled out, changing the strategies available to a planner.

The open-ended situational track showed a different pattern: most models from four of the five LLMs tended to omit expected structure rather than add it. The authors caution that the references in these open-ended cases are criteria-derived, not unique ground truths. Their scores are diagnostic and should not be read as definitive rankings of correctness.

The cultural track tested whether naming a country affected generated models. For most LLMs, it changed structural representations, while automated semantic scores shifted only slightly. That does not establish that the changes were biased or unjustified. The authors say human review of those differences has not yet been carried out; they describe the results as preliminary diagnostics, not validated findings of bias.

What the results can—and cannot—tell us

The study’s practical point is that execution checks alone cannot show whether a model captures the relevant decision problem. Structural additions or omissions could matter if downstream tools use the model to calculate reachable outcomes, reversibility, or other safety-related quantities. But this study did not measure downstream planning performance or those quantities, so it does not demonstrate that the observed differences caused particular decisions or harms.

The evidence also has important limits. The benchmark is small and hand-crafted; scenarios are symbolic and discrete rather than physical deployments. Game comparisons stop after three transitions, and the cultural conditions generally have one generation per model and scenario. The automated semantic criteria have not been validated by human annotation. Results from repeated game prompts also show variation, even with temperature set to zero, so one output should not be treated as a stable account of a model’s behavior.

The paper is a preprint, and its findings should be read as an early evaluation rather than settled evidence. The authors identify human review, longer-horizon or alternative structural comparisons, and tests of downstream planning as future work. For now, the study supports a careful but limited conclusion: a world model can be executable without closely matching a reference, and different checks reveal different kinds of mismatch.