Finding policy effects that averages can miss

Susan Athey, Maria Nareklishvili and Jann Spiess propose this approach in Debiased Conditional Independence Tests for Causal Effects, a preprint posted on November 3, 2026. Instead of predicting outcomes from treatment, their methods ask whether outcomes help predict treatment assignment beyond what researchers can already predict from observed background characteristics.

The distinction matters for evaluations involving many outcomes. The paper uses job training as a motivating example: researchers might want to know whether an intervention changes a collection of employment outcomes before investigating exactly where those changes occur. The proposed tests address that first question. They do not estimate the direction or size of the underlying effects.

Turning the question around

In observational data, participants and nonparticipants can differ before treatment. Those differences can influence both treatment assignment and subsequent outcomes. Simply predicting participation from outcomes therefore does not establish a causal effect.

The researchers instead compare the information available from background characteristics alone with the information available from those characteristics plus outcomes. Their null hypothesis is conditional independence: once observed characteristics are accounted for, outcomes provide no additional information about treatment.

Under the paper’s identifying assumptions, this corresponds to treatment leaving the entire conditional distribution of outcomes unchanged. That is more demanding than saying the average effect is zero, but less demanding than saying treatment changes nobody’s outcome. Individual gains and losses can coexist with an unchanged distribution.

Three routes to a test

The first procedure applies when treatment probabilities—the propensity scores—are known, as they may be in an experiment. It weights observations to balance treatment assignment across background characteristics, trains a treatment-prediction model on one sample, and evaluates prediction errors on another. Repeatedly drawing hypothetical treatment assignments from the known probabilities supplies the reference distribution.

The authors establish finite-sample control of false positives for this conditional randomization procedure under their assumptions. Crucially, that guarantee does not carry over simply by substituting estimated treatment probabilities.

For observational settings, they develop double-residualization tests and an augmented inverse propensity weighting test. Double residualization removes variation explained by background characteristics from both treatment and the outcome-based treatment prediction, then examines the remaining association. The augmented method combines predictions within treatment groups with weighted corrections.

These methods are designed to reduce sensitivity to certain estimation errors. They separate model training from evaluation through sample splitting. Their implementation also adds a small independent normal random perturbation because the ideal scores have zero variance under the null, making ordinary standardization problematic.

Where the simulations help—and where they do not

The main simulations use samples of 500, 1,000 and 2,000 observations, with five, ten or thirty outcomes. They compare the proposed procedures with a joint Wald test based on a system of linear outcome regressions.

The gains are substantial in some designs where positive and negative effects largely cancel. In the “No Baseline” design with 1,000 observations and five outcomes, the augmented test detects effects in 98.4% of simulations, compared with 5.5% for the Wald test. Additional experiments examine heavy tails, outliers and mixtures of outcome distributions.

But the results do not establish universal superiority. With 500 observations and five outcomes in the “Full Heterogeneity” design, the Wald test detects effects in 98.7% of simulations, versus 82.6% for the augmented test. The authors attribute this advantage to detectable mean shifts and the estimation costs of flexible models. Comparisons with kernel and generalized covariance tests likewise show complementary strengths in small samples.

The limits of the causal claim

The causal interpretation assumes that observed characteristics account for selection into treatment: conditional on them, assignment must be independent of potential outcomes. The methods do not remove bias from unobserved confounding. The setup also assumes independent observations and no spillovers between individuals.

Most procedures require treatment probabilities bounded away from zero and one. A residualization variant relaxes this requirement, but can lose sensitivity when distributional differences concentrate among people almost certain to receive, or not receive, treatment.

The observational tests have large-sample rather than finite-sample validity, with conditions on how accurately supporting models are learned. Their false-positive rates can be elevated: one main simulation reports 14.8% for the augmented test at a nominal 5% level. Poor treatment prediction can also reduce power.

The evidence is methodological and simulated, not a new evaluation of an actual policy. The preprint offers a way to search for distributional effects that averages may conceal—not a guarantee that a detected association is causal, or that an undetected effect is absent.