When treatments spill over, an experiment may answer the wrong question

Random assignment helps researchers distinguish treatment effects from pre-existing differences. But what happens when a participant’s outcome also depends on whether other participants receive treatment? This complication, called interference, arises in settings involving peer influence, vaccination and network experiments.

A preprint by Vishesh Karwa, Edoardo M. Airoldi and Donald B. Rubin argues that researchers must distinguish two questions: what treatment does under specified neighborhood conditions, and what it does on average under a particular allocation policy. Their mathematical results show why ordinary randomization can make the first question dramatically harder to answer than the second.

Three assumptions to keep separate

The authors begin with potential outcomes: the outcomes each participant would have under different treatment assignments. With interference, every assignment across the population could produce a different outcome for each person. The number of possibilities grows exponentially with population size.

They organize simplifying assumptions into three layers. An interference neighborhood specifies whose treatment can affect a participant. An exposure mapping summarizes those assignments, perhaps by counting treated neighbors. An outcome model adds restrictions, such as assuming that exposure has a linear effect.

These are different commitments. Assuming that only the number of treated neighbors matters, rather than their identities, is already a substantive restriction—not merely a convenient way to display data.

Once neighborhoods and exposure mappings are specified, the authors show that outcomes can be represented through an own-treatment component, a spillover component and an interaction between treatment and exposure. This decomposition is an accounting identity, not an assumption that responses are linear in exposure. Setting the interaction to zero would be an additional assumption.

Two meanings of a direct effect

The paper’s fixed-condition direct effect compares treatment with control while holding all relevant neighbors untreated. Its total effect compares an entirely treated neighborhood, including the participant, with an entirely untreated one. These are examples of what the authors call average causal effects.

An expected direct effect instead compares treatment with control while averaging over the neighborhood exposures generated by a specified allocation policy. The policy might assign each participant treatment independently with a chosen probability.

Neither question is inherently better. They answer different scientific questions. The allocation policy defining an expected effect is also conceptually distinct from the experimental design used to estimate it, even when the two coincide.

The direct effects coincide if treatment and exposure combine additively. Otherwise, the policy-averaged direct effect includes a treatment–exposure interaction that the fixed, unexposed comparison excludes.

Why some observations become scarce

The researchers analyze Horvitz–Thompson estimators, which weight observed outcomes by their probabilities of being observed. Their variance bounds distinguish three sources of uncertainty: weighting rare observations, overlapping interference neighborhoods and additional dependence created by the assignment design.

Consider independent treatment assignment with probability one half. For a treated participant to have every relevant neighbor untreated, each extra neighbor halves the probability of that condition. Observations needed for the fixed direct effect therefore become exponentially rarer as neighborhood size grows.

Without additional outcome restrictions, observations at other exposures cannot substitute for those missing conditions. Lower bounds show that this obstacle is not simply a defect of the particular weighted estimator: it constrains any estimator required to be unbiased across all bounded outcome tables in the specified model.

For policy-averaged effects aligned with the assignment design, the weighting problem can be much milder. Network dependence still matters, however. A hub can connect many outcomes, so a small average number of neighbors does not guarantee precise estimation.

Designing for the question

The authors construct graph-aware alternatives. To estimate the fixed direct effect, they select mutually nonadjacent focal participants, randomize their treatment and hold their neighbors untreated. For the total effect, focal participants must be farther apart so their immediate neighborhoods do not overlap; each focal neighborhood is then assigned entirely to treatment or control.

Under the stated assumptions, these constructions replace exponential dependence on maximum neighborhood degree with polynomial variance bounds. That is not a promise of precision on every network. The bounds still depend on network size and structure. Cluster randomization can also help, but neighborhoods split across many clusters can retain the rare-observation problem.

Simulations using fixed synthetic outcomes and randomly generated networks illustrate these patterns. They are not evidence of treatment effects in real populations. The work is an arXiv preprint, and its guarantees depend on bounded outcomes and appropriate interference assumptions. An observed social network need not capture every causal connection.

The practical lesson is to choose the effect and the assignment scheme together. Otherwise, an experiment may estimate a policy-averaged effect reasonably well while providing little information about the fixed-condition effect researchers intended to measure.