Good Predictions Are Not Enough: Testing What Economic Models and AI Rule Out

A second question for prediction

When a model predicts well, researchers still need to ask what it would struggle to explain. A procedure capable of fitting many different relationships offers a different kind of explanation from one that fits observations while ruling out substantial alternatives.

In their version-four preprint, Drew Fudenberg, Wayne Yuan Gao and Zhiheng You extend a measure called restrictiveness to prediction procedures whose possible outputs cannot easily be listed. Their approach evaluates the entire workflow, including estimation, model selection, prompts, computational limits and failure handling—not just a named model or its parameter count.

Testing against simulated alternatives

The researchers generate alternative prediction rules satisfying selected economic restrictions. They supply each procedure with noise-free observations from those rules, then evaluate its predictions on separate inputs. Procedural restrictiveness is its average discrepancy from these simulated truths divided by the corresponding average discrepancy of a baseline.

A low score means the procedure approximates the chosen alternatives well on average. A high score is more ambiguous: it can reflect genuine restrictions on possible predictions, but also limited learning, unsuccessful optimization or execution failures. Removing observation noise does not remove these implementation constraints.

The authors pair this measure with completeness, which evaluates prediction on observed data. Completeness expresses a procedure’s improvement over a baseline as a fraction of the improvement achieved by a flexible benchmark. It is not a percentage of predictions that are correct, and it can exceed one or become negative.

The paper also establishes a conditional theoretical connection: as training samples grow, procedural restrictiveness converges to model-class restrictiveness if both the fitting procedure and its baseline consistently attain their respective classes’ best approximation performance. Such guarantees are generally unavailable for the AI workflows studied.

Lottery predictions: similar fit, different flexibility

The first application predicts certainty equivalents: sure amounts valued the same as risky lotteries. The empirical comparison uses 4,459 responses from 179 participants evaluating 25 binary lotteries. Simulated alternatives are constructed using Gaussian processes, with prize bounds and monotonicity restrictions imposed at sampled lotteries.

Two workflows using Claude Sonnet 5.5 perform almost identically on observed responses. Completion generates prediction code without execution feedback at low effort; Agent uses high effort and can execute Python for up to 20 rounds. Their completeness scores are 0.988 and 0.989, respectively, relative to the TabPFN benchmark.

Yet with 100 simulated training observations, Completion’s restrictiveness is 0.328, compared with Agent’s 0.025. Agent can accommodate a much broader range of the simulated relationships. This comparison concerns bundled workflow choices, including reasoning effort and execution access, rather than isolating the effect of tool use alone.

A two-parameter cumulative prospect theory specification provides another contrast. Its completeness is 0.976, identical to the full three-parameter specification, but its restrictiveness is 0.584 rather than 0.123. In this setting, removing one parameter preserves empirical performance while imposing substantially more structure. Prompt changes also alter AI restrictiveness while leaving lottery completeness nearly unchanged.

Market shares are not price responses

The second application examines demand for 24 differentiated products. The authors fit procedures to simulated market shares, then separately assess their share predictions and implied price elasticities—how demand responds to changes in a product’s own price or competitors’ prices.

At 100 training markets, share restrictiveness falls from 0.183 for multinomial logit to 0.173 for nested logit and 0.140 for mixed logit. Nevertheless, nested logit performs worse on both elasticity measures, while mixed logit has worse cross-price elasticity performance and similar own-price performance. Better approximation of shares does not automatically recover better substitution patterns.

Agent and TabPFN are less restrictive than these fitted structural procedures on all three criteria at that sample size. On observed cereal-market data, Agent reduces prediction error by about 30% relative to the baseline, compared with 26% for TabPFN. However, both can exploit product-specific effects omitted from the structural specifications, limiting a like-for-like interpretation.

What the comparison can establish

These results depend on the eligible rules, their sampling distribution, the discrepancy being measured and the implemented workflow. Sensitivity exercises preserve several central comparisons, but some elasticity rankings change under reweighting. Finite-point lottery constraints do not establish restrictions throughout the continuous domain.

The empirical datasets are public, so prior exposure could contribute to language-model performance. Observed cereal shares also do not directly reveal true price elasticities; the elasticity comparisons rely on simulations. The authors’ central message is therefore not that flexibility is inherently good or bad, but that predictive fit alone leaves an important question unanswered: how much does the procedure actually rule out?