A Statistical Test for Where Treatment Effects Vary

A policy can have a positive average effect while helping some groups more than others. Researchers often want to know not only whether the effect varies, but also which characteristics are associated with that variation. Answering the second question is difficult when characteristics such as age and income are related, or when their effects interact.

In a preprint, economist Julius Owusu develops statistical tools for examining this problem. The paper proposes tests that compare estimated treatment effects across values of one characteristic while holding another fixed. It also develops measures of the size of that variation, expressed in the same units as the outcome being studied.

Why compare two points?

The object of interest is the conditional average treatment effect: the average difference an intervention would make for people with specified characteristics. It is an average within a group, not a claim that every individual in the group responds identically.

One way to examine variation by income is to average over age and see whether the resulting effect changes with income. But averaging can hide a pattern. If an effect rises with income at some ages and falls with income at others, those changes can cancel. Correlation can also mislead: an apparent income pattern could reflect variation associated with age instead.

Owusu's approach first estimates treatment effects jointly over both characteristics using kernel smoothing, a method that combines nearby observations to estimate a function without prescribing a particular shape. It then compares estimates at pairs of income values while holding age fixed, and aggregates the absolute differences across the chosen region. The same procedure can be used in the other direction. A separate version uses one-sided differences to test a monotonicity restriction.

The pairwise structure creates a statistical problem: comparisons can be dependent even when their other evaluation points are far apart, because they share information at one point. The paper develops a large-sample normal approximation using a temporary randomization of sample size, known as Poissonization, and a projection that separates the leading contribution into single-point terms. It also proposes corrections for the apparent differences created by estimation noise and an estimator of uncertainty that accounts for overlapping smoothing windows.

Simulated tests and a retirement-saving example

In the main simulations, the paper reports 1,000 repetitions per design with 1,000 observations. When both directional null hypotheses were true and treatment probabilities were known, the corrected test rejected at least one of them in 5.3% of repetitions, close to its nominal 5% familywise level. In a simulated interaction that disappeared after averaging over either characteristic, it correctly rejected both directional nulls in 84.3% of repetitions. Other methods performed better in some designs and worse in others; the results depend on the chosen models and implementations.

The supplementary simulations also show limitations. At 500 observations, the corrected test rejected too often in some null settings. Some simulation windows extended beyond the data support, introducing boundary bias that the proposed noise corrections do not remove. The paper also notes that its normal approximation has not been proved for grids whose spacing remains a fixed fraction of the smoothing bandwidth.

For an empirical illustration, Owusu analyzes a 1991 survey extract containing 9,915 households. The treatment is eligibility for an employer-provided 401(k), not participation itself, and the outcome is net financial assets. The analysis compares estimated eligibility effects across income at fixed age and across age at fixed income, adjusting for other observed household characteristics.

Using the main specification and adjusting for the two tests, the paper reports evidence of variation in both directions. The income result was stronger: a noise-corrected, density-weighted measure of income-related variation was approximately $2,704. The corresponding age measure was about $1,369, but its magnitude was not distinguishable from zero. This does not contradict the age test: the paper's magnitude interval is constructed differently and is not an inversion of that test.

What the results do not establish

The 401(k) data are observational. Interpreting the estimates as causal effects requires, among other assumptions, that eligibility is unrelated to potential outcomes after adjustment for the included characteristics. The paper notes that this assumption is contested. Its main propensity model is also rejected by specification tests, while the outcome models are not considered credible as exact models. A split-sample analysis with a more flexible first stage supported the income result but did not consistently confirm the age result.

Further uncertainty comes from very heavy-tailed assets and boundary effects in some estimation windows. The theory is designed for low-dimensional characteristic blocks under smoothness and other regularity conditions; extending it to higher dimensions and cross-fitted nuisance estimates remains open. A failure to reject a null should not be read as proof that the effect is constant. The contribution is a way to test and quantify conditional variation under stated assumptions, rather than a guarantee that those assumptions hold in a particular policy setting.