When Different Differences Tell Similar Stories

A policy arrives in different places at different times. To estimate its effects, an economist might compare how outcomes change in early adopters with changes in places that adopt later or never adopt at all. This is the intuition behind staggered difference-in-differences, a widely used approach to studying policies and other interventions.

The complication is that a popular implementation, two-way fixed effects (TWFE), can combine treatment effects in surprising ways. When effects differ across places or over time, its implicit weights can be negative. In theory, the resulting estimate can be negative even if every individual treatment effect is positive. That possibility has helped motivate a new generation of estimators.

But a theoretical possibility is not a measure of frequency. In the preprint How Different are Different Differences-in-Differences?, Peter Hull examines how often conventional and modern estimates diverge in a large collection of economics research. His answer is reassuring in some respects, but it is not an all-clear for any particular study.

A large comparison, with boundaries

Hull used AI-assisted screening and replication to search 16,094 records from ten economics journals. The final sample contains 1,430 eligible, replicated specifications from 116 papers. Human judgment, oversight and spot-checking remained part of the process; in some cases, original authors supplied additional materials.

The study focuses on relatively simple staggered-adoption designs with binary treatments. It excludes, among other complications, specifications with time-varying controls or unit-specific trends. Some comparisons were also dropped because high leverage could make estimates unstable or inference unreliable. The resulting evidence is broad within those boundaries, not a census of all difference-in-differences research.

Hull compares TWFE with two alternatives: DiD-averaging, which combines cleaner group-and-time comparisons using nonnegative weights, and imputation, which estimates untreated outcomes and compares them with observed outcomes for supported treated observations. Both still rely on identifying assumptions, including versions of parallel trends and no anticipation.

Agreement is common, not universal

Across the retained comparisons, TWFE estimates correlate at 0.89 with DiD-averaging estimates and 0.93 with imputation estimates. Their signs differ in 14.9% and 12.0% of comparisons, respectively. A smaller share—3.9% and 2.8%—both changes sign and has a statistically significant difference between the two estimates. These comparisons include some pretreatment event-study summaries as well as treatment-effect estimates.

The preprint also examines the weights directly. Among static TWFE specifications, the median share of treated observation rows with negative weights is about 8%; the median share of absolute effect weight carried by those negative weights is just 1.4%. That helps explain why the theoretical problem may have a limited footprint in a typical specification, although the distribution varies and some designs have much larger negative-weight shares.

There is a countervailing consideration: precision. On average, standard errors are 40% larger for DiD-averaging and 9% larger for imputation than for their TWFE counterparts. In settings where the estimands are close, a more robust method may therefore produce a noisier estimate. That is a tradeoff to investigate, not proof that precision is more valuable than robustness.

What noise can—and cannot—explain

Observed estimates can disagree simply because they are noisy. Hull uses a hierarchical model to separate sampling error from differences in the underlying quantities targeted by the estimators, known as estimands. In the baseline model, estimated correlations between TWFE and modern estimands are 0.91 for DiD-averaging and 0.97 for imputation.

Across the sampled comparisons, the model estimates a 2.4% probability for DiD-averaging and a 0.7% probability for imputation that both the TWFE estimate is statistically significant and its sign is opposite to the corresponding modern estimand’s sign. These are joint probabilities, not probabilities conditional on TWFE significance. They are also model-based quantities, not the observed sign-flip rates or guarantees for a new study. An alternative nonparametric model estimates the same joint probabilities at 2.4% and 1.8%, showing that the imputation figure is sensitive to modeling choices.

Nor does disagreement isolate negative weights perfectly. Different support or aggregation, failures of parallel trends or no anticipation, and sampling error can also matter. Agreement between methods does not establish that either has recovered a causal effect.

A reason to check, not to relax

The results suggest a more nuanced response than either abandoning TWFE everywhere or treating negative weights as harmless. Hull finds that larger always-treated shares are associated with more sign reversals for DiD-averaging, while larger never-treated shares are associated with fewer reversals for imputation. Those associations are consistent with the theory, though they do not establish a causal explanation for every difference.

The practical lesson is to report and investigate the comparison. A modern estimator can reveal whether a conclusion depends on problematic weighting; the TWFE estimate can still illuminate the precision cost of that robustness. The preprint’s broader contribution is to put empirical scale beside theoretical warning—while leaving researchers responsible for examining the design in front of them.