When spillovers make ignoring A/B test data the safest static estimate

A test of estimation under spillovers

A theoretical preprint by Arthur Delarue of UVA Darden and Ilan Lobel of NYU Stern identifies circumstances in which ignoring experimental observations is optimal under a worst-case error criterion. The finding concerns a specific class of estimators and a mathematical model of interference, not evidence that A/B testing generally fails.

Their paper, Minimax-Optimal Static Estimators under Interference, studies the global treatment effect: the difference between total outcomes when everyone receives a treatment and total outcomes when nobody does. That differs from comparing treated and untreated participants within an experiment, where each group may influence the other.

The distinction matters for digital platforms whose users interact. The authors ask what an analyst can reliably estimate without committing to a detailed model of those interactions.

Guarding against the worst allowed case

The model represents participants as nodes in an undirected network. Each has the same number of neighbors, denoted by d, and the network looks structurally identical from every participant’s position. Complete networks and cycles are examples.

Each participant independently receives treatment with probability p. Outcomes must remain between −1 and 1. A parameter called delta caps how much changing one neighbor’s assignment can change a participant’s outcome; non-neighbors have no influence under the model.

Rather than select one outcome-generating process, the researchers allow an adversary to choose any process satisfying those restrictions. They seek an estimator that minimizes the largest possible mean-squared error, averaging over the experiment’s random assignments. This is the minimax criterion.

Crucially, the search is limited to static linear estimators. Each observed outcome receives a fixed weight determined by the participant’s identity and their own treatment status, not by the full pattern of assignments. Network symmetry lets the authors reduce the optimization to one common treatment weight and one common control weight.

Why zero can be optimal

For balanced assignment, with each participant treated with probability one-half, the paper establishes a sharp threshold: when delta reaches 2/d, reporting zero regardless of the observations is minimax-optimal among static estimators.

The threshold has an intuitive interpretation. With outcomes bounded between −1 and 1, a participant’s largest possible treatment effect is 2. At delta equal to 2/d, the combined influence of all d neighbors can match that range.

A five-participant example illustrates the danger. Everyone is connected to everyone else, and outcomes depend on the number of treated neighbors. Moving from no treatment to universal treatment reduces each outcome from 1 to −1, producing a total effect of −10. Yet whenever both experimental groups are present, treated participants have outcomes one-half point higher than control participants. The observed between-group comparison points in the opposite direction from the global effect.

Reporting zero does not establish that the treatment has no effect. It is a defensive choice under the specified loss function: no static estimator can achieve a smaller worst-case mean-squared error in this regime. The zero estimator’s worst-case error is itself large, at 4n² for n participants.

Unequal assignment changes the response

When treatment probabilities are unequal, the model supports an intermediate strategy: ignore the group that is smaller in expectation and use only outcomes from the larger group. The authors call this a “half-zero” estimator.

For a fixed treatment probability different from one-half, they prove that such an optimal estimator exists for sufficiently large populations once interference exceeds a threshold with a finite-sample adjustment. The leading threshold is 1 divided by d times the larger assignment probability.

A separate condition determines whether discarding both groups becomes optimal. Under sufficiently unequal assignment, the zero estimator never becomes minimax-optimal, even at the largest effective interference levels. Thus, the balanced experiment’s complete retreat from estimation does not automatically carry over to unbalanced experiments.

What the results do—and do not—recommend

Even without interference, the analysis produces a refinement. For balanced assignment and finite populations, shrinking the conventional Horvitz–Thompson estimator toward zero improves worst-case error. The optimal multiplier is √n/(√n + 1), which approaches one as the population grows.

With strong interference, the authors’ large-sample risk bounds favor increasingly unequal assignment. Their interpretation is to measure one global state directly and use outcome bounds to constrain the unobserved alternative. These bounds do not establish that every finite network benefits from the same redesign.

The contribution is mathematical, not an empirical demonstration on a platform. Its warning is specifically about static estimators under the stated uncertainty set. It does not establish that estimators using richer interference assumptions, or alternative experimental designs, must also fail.