Can statistical ‘digital twins’ help evaluate job training?

Evaluating job training requires more than checking whether graduates earn more afterward. Their earnings might have risen anyway, or enrollment might follow a temporary job loss. Randomized controlled trials address this problem by comparing people randomly offered a program with a control group. But an experiment at one location and time does not necessarily describe what the program achieves elsewhere or years later.

In this preprint, Raj Chetty and colleagues ask whether large administrative datasets can support a complementary approach: statistical “digital twins” that estimate participants’ earnings without a program. These are predicted counterfactual outcomes, not simulated people or exact matches to individual non-participants.

Building the missing comparison

The researchers link workforce-program records covering 105,216 participants to anonymized federal tax data. Their population dataset covers 327 million people who were adults at some point between 1999 and 2025. Analysis takes place within the Census Bureau’s secure research environment, with reported statistics reviewed for confidentiality protection.

The models use information available before enrollment, including five years of earnings and employment history, prior industries, demographics and residential characteristics. The earnings measure is annual W-2 wages, expressed in 2025 dollars; it excludes self-employment income.

Gradient-boosted decision trees predict earnings among non-participants. A second model estimates the probability of program participation. The researchers combine them using double/debiased machine learning, which corrects for certain prediction biases introduced by model regularization. It does not automatically remove selection bias from unobserved characteristics such as motivation.

For their baseline estimates, the researchers use a random 10% subsample of eligible non-participants after applying age, earnings and geographic restrictions. Experimental control groups are excluded from model training and tuning, preserving their role as an independent benchmark.

Agreement is not the same as no bias

The validation sample contains 37 randomized trials spanning 24 distinct training and employment programs. The principal outcome is mean annual earnings two to five years after enrollment. Observational and experimental estimates use the same treatment group, but construct their untreated comparison differently.

Across trials, the two sets of estimates have a correlation of 0.93. Yet the researchers reject the hypothesis that the observational estimates are unbiased in every trial. Their argument is therefore not that machine learning eliminates confounding, but that the remaining errors may be small enough for useful decisions.

After accounting for sampling noise in the experimental and observational counterfactuals, they estimate a root mean squared bias of $1,717. This measures the typical magnitude of remaining bias without allowing positive and negative errors to cancel.

The headline accuracy comparison needs another qualification. At the treatment-group sizes used in the trials, total observational root mean squared error is $2,257, compared with $2,126 for the randomized estimates. Adjusting for the larger participant samples typically available in their observational evaluations reduces the observational estimate to $2,073. These estimated errors are comparable—not evidence that the method universally beats randomization.

The authors also find that neither a larger dataset nor a flexible algorithm is sufficient alone. Rich work histories, additional covariates, large samples and flexible earnings models work together to improve performance.

Extending evaluation beyond a trial

Applying the method outside experimental enrollment years, the researchers estimate substantial changes in Year Up United Core’s effects. Estimated annual earnings gains two years after enrollment rise from $5,198 for the 2007 cohort to $12,000 for the 2017 cohort, then fall to $9,077 for 2022. Effects also differ across locations.

Assessing uncertainty in these new settings requires an explicit assumption: their observational biases must share the same marginal distribution as biases in the validation trials. That is more plausible for programs with similar participants, selection processes and structures. The validation trials are not a random sample of all workforce programs, and the authors do not establish equivalent reliability in other policy domains.

Within Year Up United, higher target-sector job-placement rates are associated with larger estimated earnings gains. The paper explicitly cautions that this does not establish placement as the cause: locations may differ in participant selection, labor demand or implementation quality.

The practical proposal is to use experiments to calibrate observational evaluation, then update estimates across sites and cohorts. Its promise rests on measured—not absent—bias, and on recognizing where experimental validation stops.