When Spillover Tests Shape an Experiment’s Next Question

An experiment’s preliminary checks can change its main analysis. Finding evidence that a treatment affects people beyond its recipients may lead researchers to study spillovers; finding no such evidence may steer them toward direct effects alone. A new preprint develops statistical tests that explicitly account for this fork in the analytical road.

Why a preliminary test matters

Randomized experiments do not always separate neatly into treated people and unaffected controls. One person’s treatment can influence another person’s outcome, a phenomenon called interference or treatment spillover. Researchers may therefore test for spillovers before deciding how to analyze an intervention.

That creates a second statistical issue. If an initial result determines the next model or hypothesis, the subsequent analysis has been selected using the data. A test that behaves correctly across all possible experiments need not behave correctly within just the experiments that pass through a particular analytical branch.

In Selective Randomization Inference for Interference, Kyoto University economist Takahide Yanagi develops a procedure that accounts for this selection. The work is an arXiv preprint, and its guarantees depend on the experimental setup and assumptions specified in the paper.

Two branches, explicitly accounted for

The paper focuses on randomized saturation designs. Researchers first randomly assign clusters different target treatment proportions, then randomly select treatment recipients within each cluster. These are two stages of treatment assignment, distinct from the method’s two stages of statistical testing.

The first statistical test asks whether each person’s outcome depends only on their own treatment, rather than anyone else’s assignment.

If this no-spillover hypothesis is rejected, the second test compares two pre-specified exposures. Exposure combines a person’s own treatment status with their cluster’s assigned treatment proportion. Comparing treated and untreated people at the same proportion tests direct effects; comparing untreated people across different proportions tests spillover effects.

This branch assumes that the exposure description is correct: assignments producing the same exposure must produce the same outcome for that person. Rejecting the no-spillover hypothesis does not itself establish that assumption.

If the first test does not reject, the main procedure instead assumes no spillover and tests for direct effects. That assumption remains necessary; a non-rejection is not proof that spillovers are absent. The paper also explains how researchers could retain an exposure-based analysis allowing spillovers after a non-rejection, provided they account for that selection.

Reconstructing the decision

Randomization inference compares an observed statistic with statistics from alternative assignments allowed by the experimental design. Interference complicates this because the hypotheses being tested do not generally reveal every outcome that would occur under every alternative assignment.

Yanagi addresses this by selecting “focal” groups of participants and restricting the alternative assignments. In the first test, focal participants keep their observed treatment status. Under the no-spillover hypothesis, their outcomes can then be inferred even as other assignments change.

The second test requires more than reconstructing its own focal participants’ outcomes. It must also reconstruct the earlier test’s decision under alternative assignments, using the relevant second-stage hypothesis rather than simply assuming the first hypothesis was true.

The selective test then compares assignments that reproduce the observed first-stage decision while satisfying the other conditioning restrictions. Under the stated conditions, Yanagi proves that each second-stage test controls its selective Type I error rate in finite samples. In plain terms, among experiments reaching the relevant branch and sharing the specified conditioning information, the probability of rejecting a true second-stage null is bounded by the chosen significance level.

Simulations and a cash-transfer illustration

The main simulations use 40 clusters of three participants and 10,000 replications per parameter configuration. At a nominal 10% significance level, reported second-stage false-positive rates under the relevant assumptions are close to 10%. Detection rates increase as simulated effects become larger.

Additional simulations show why selection matters. A conventional exposure-contrast test that ignored the first-stage decision rejected a true null in 16.8% of selected experiments in one configuration, despite a nominal 10% level. Ignoring selection can also make tests conservative rather than overly permissive.

The empirical illustration reanalyzes part of a Malawi cash-transfer experiment: 334 baseline schoolgirls across 18 enumeration areas, with outcomes observed for 322. None of four outcome-specific spillover tests rejects at the 10% level. Subsequent selective tests, assuming no spillover, reject the no-direct-effect hypothesis only for whether a girl had ever married, with a p-value of 0.076.

Under the additional assumption of homogeneous direct effects, the reported 90% selective confidence interval for that outcome ranges from −18.7 to −0.7 percentage points. Yanagi cautions that the analysis uses only a subset of the original experiment.

The method also involves practical tradeoffs: more focal participants can restrict the available assignments and reduce power, while general selective computations can be demanding. Its contribution is therefore specific: accounting for a spillover-driven analytical choice, not eliminating the assumptions needed to interpret the chosen analysis.