Choosing Causal Comparisons by Looking Forward, Not Just Back

The missing outcome behind every comparison

Evaluating a training program involves an unavoidable gap: researchers can observe participants’ earnings after the program, but not what those same people would have earned without it. In observational studies, a comparison group supplies an estimate of that missing outcome.

Researchers commonly choose controls who resemble participants in measured characteristics, or compare groups with similar earlier outcome patterns. In his preprint, Sapienza University of Rome researcher Augusto Cerqua proposes giving another criterion a central role: how similar are their predicted outcomes without treatment?

The proposal builds on existing work on prognostic scores, which summarize information about expected outcomes. Its contribution is a framework for using those predictions to construct comparisons, with different prediction targets for different causal designs.

Forecasts choose controls; observations measure effects

For a comparison of post-program earnings levels, the relevant score predicts earnings without the program. For difference-in-differences, which compares changes over time, the relevant score predicts the untreated change from the last pre-program observation.

This distinction matters because people with different starting earnings could nevertheless experience similar earnings changes without treatment. Conversely, similar personal characteristics need not imply similar future outcomes.

Cerqua considers two ways to build the forecasts. One uses evaluation-period outcomes from untreated controls, together with their earlier characteristics. The other learns exclusively from historical information and projects that relationship forward. The historical approach keeps evaluation-period outcomes out of comparison-group construction, but requires sufficiently stable predictive relationships over time.

In both cases, the forecasts select or weight controls. Actual observed outcomes—not predicted outcomes—then determine the treatment-effect estimate.

Why accurate predictions are not enough

The paper derives an exact breakdown of estimation error into two parts: imbalance in the relevant prediction score, and differences in average forecast errors between participants and weighted controls.

Balancing the score removes the first part, but not necessarily the second. A model could predict controls well while systematically misjudging what participants would have experienced without treatment.

An unexpected shock affecting both groups equally can cancel when their observed outcomes are compared. A shock affecting the groups differently does not. Thus, neither predictive accuracy nor score balance alone establishes a valid causal comparison. The approach does not replace the assumptions underlying matching or difference-in-differences.

A test against an employment experiment

Cerqua applies the framework to the National Supported Work demonstration, a US subsidized employment and training program from the 1970s. The analysis uses 185 treated men and 15,992 nonexperimental controls from the Current Population Survey. Another 260 randomized controls provide experimental benchmarks; they are excluded from prediction and observational comparison-group construction.

The forecasts use TabPFN 3.5, a tabular foundation model pretrained on synthetic prediction tasks. Three-fold cross-fitting ensures that no control’s evaluation outcome is used to generate its own prediction. Controls are then selected through nearest-neighbor matching or weighted through entropy balancing, which matches the first two moments of the chosen score.

For earnings levels, the experimental benchmark is $1,794.34. Matching on predicted untreated earnings produces $1,651.17, the closest level estimate among the observational procedures examined. Conventional propensity-score matching, which summarizes the likelihood of treatment participation, produces $1,990.77.

For difference-in-differences, the benchmark is $1,529.20. Entropy balancing on predicted untreated earnings levels produces the closest estimate, $1,468.45. Notably, balancing predicted changes—the target more directly aligned with this design—does not get closer in this application.

Promising results, bounded conclusions

The historical-only forecasts perform less well. Their implementation learns the 1974–1975 earnings relationship and applies it to forecast 1976, using that forecast as a proxy for untreated earnings in 1978. This mismatch between forecasting and evaluation horizons limits what the results establish about historical forecasting more generally.

The application also challenges a simple reading of balance diagnostics: propensity-score matching achieves much better measured covariate balance than matching on predicted earnings, yet its level estimate is farther from the experimental benchmark. Closeness to that benchmark, however, can reflect sampling variation as well as differences in bias.

Cerqua presents the exercise as a proof of concept, not evidence of a universally superior method. The experimental benchmarks themselves carry sampling uncertainty, and reported observational standard errors exclude uncertainty from estimating scores and constructing weights. TabPFN’s prediction improvements over ordinary least squares are also modest here.

The practical message is therefore specific: forecasts can help researchers target the missing untreated outcome more explicitly. They cannot make that unobserved outcome observable, or turn a well-balanced comparison into causal proof.