Unsatz tests whether AI can choose a scientific model—not just fit one

A model can match the measurements already collected and still misunderstand the system that produced them. A new preprint by Ethan Ji, Wei Wang and Yizhou Sun at the University of California, Los Angeles, makes that distinction central to evaluating AI-assisted scientific discovery.

Their benchmark, Unsatz, asks language-model agents to investigate simulated systems without being given their governing equations or a complete inventory of their hidden states. Across 80 systems in 20 domains, the nine evaluated models solved between 17 and 48 systems. The authors argue that choosing a model’s structure remains a substantial obstacle, even when software handles coefficient estimation.

Choosing what the model contains

The name refers to an ansatz: a proposed mathematical form. Here, that means deciding which variables exist, which are hidden, how they interact and how instruments turn them into measurements.

Consider the benchmark’s ecological example. An agent sees two population signals and can change starting populations, adjust nutrient replenishment or perturb the system. What it does not initially know is that two hidden resource stores help govern the populations. Explaining the observations requires more than fitting a relationship between the two visible signals.

Unsatz includes ordinary and delay differential equations, spatial systems, discrete updates, static and causal relationships, and stochastic processes. Agents receive experimental controls, prices, budgets and a modeling language. Some recorded channels may be uninformative or redundant, while others may distort the underlying states.

The search is not unrestricted: the language provides a fixed mathematical vocabulary and generally allows at most 16 fitted coefficients, including hidden-state initial values. Every generated system is expressible in that language. Agents must assemble a suitable structure within those constraints.

Paying for evidence, predicting beyond it

Experiments consume budget. Depending on the system, agents can change inputs, perturb or hold states fixed, purchase wider-range or longer runs, or obtain an exact reading of a hidden state. They propose executable models, while a shared fitter estimates coefficients from purchased observations. Agents cannot run their own code; their quantitative feedback comes from the benchmark’s tools.

Only the last submitted model determines the final score. Hidden tests examine unfamiliar conditions, interventions and longer runs, with details varying by domain. For deterministic systems, success requires the worst test-group normalized error to be below 0.1. Stochastic systems are judged using distributional scores rather than whether individual random trajectories match.

Crucially, agents never see these hidden scores during their investigations. A low error on purchased data therefore does not tell them whether their explanation will generalize.

Structure proves harder than coefficient fitting

The reported leader, GPT 6 Astra, solved 48 of 80 systems, or 60%. Claude Fable 5.1 solved 38. The remaining seven models solved between 17 and 24 each.

Classical comparisons covered a smaller subset. Across 76 applicable method–system pairings on 40 deterministic systems, SINDy and the authors’ grammar search recorded no successes, while PySR solved three systems. Given the true structure and the same randomly acquired data, the fitter solved 27 of those 40 systems, plus three of four stochastic systems.

That comparison supports the authors’ structural-bottleneck interpretation, but it is not a perfectly matched contest: language models selected their experiments, whereas classical methods used random acquisition designs. Correct equations also did not guarantee successful fitting from random coefficients.

Nevertheless, expanding the fitting effort for 248 executable but failing final deterministic models produced no additional passes.

Finding variables is not finding mechanisms

Agents sometimes identified hidden-state counts or interaction graphs without discovering adequate equations. In one twelve-node network, two agents recovered all sixteen directed connections. Only the model with suitable nonlinear dynamics passed the predictive threshold; the linear model failed.

Revision could also make matters worse. Among 510 episodes with multiple submissions, 235 ended with a model that predicted worse than an earlier submission. In one ecology case, a revision reduced training error by 76% while its hidden-test error rose from 0.29 to 126.

Transfer experiments offered a narrower positive result. In six studies where an agent solved the first system, supplying its discovered program helped a fresh instance solve five related systems, compared with two without that program. Neither approach solved the corresponding controls lacking the shared mechanism. The small sample limits the conclusion.

Unsatz measures discovery within generated, constrained worlds—not discovery in an unrestricted laboratory. Its budget calibration also assumes access to the true structure and favorable coefficient initialization. Within that setting, the preprint highlights a consequential distinction: identifying the ingredients of a model, or improving its fit, does not establish that it captures how a system behaves.