When a Different Statistical Method Changes the Direction of a Finding

The problem with a familiar comparison

What happens to sales when a platform introduces a new feature, or to firms when a regulation takes effect? Researchers often cannot randomly assign such changes. Instead, they may use a method called difference-in-differences: compare how an outcome changes in an affected group with how it changes in a comparison group.

The design becomes more complicated when different groups experience the change at different times. A common regression approach, known as two-way fixed effects, accounts for stable differences between groups and for changes shared across calendar periods. But when treatment effects vary across groups or over time, it can implicitly assign negative weights to some effects. Its overall estimate can then fall outside the range of the underlying effects and even have the opposite sign.

In a preliminary working paper, Davide Proserpio asks how much this concern matters for published business research. Rather than assuming that a theoretical problem necessarily overturns findings, he reanalyzes available studies using two methods designed to handle varying treatment effects.

Revisiting published analyses

The study focuses its re-estimation on papers in Management Science and Marketing Science with usable replication data. Language-model agents screened articles and replication materials under a fixed protocol, and additional agents audited records against the available packages and articles. The paper says judgment calls were logged. The agents also helped replicate and re-estimate the studies.

The main sample contains 355 eligible specifications from 40 papers. A specification is a particular analysis, which might differ from another in its outcome, sample or treatment definition. Some papers contribute many specifications, so these are not 355 independent studies.

For each eligible design, the reanalysis compares the conventional estimate with two alternatives. The first estimates effects for groups treated at different times and averages those effects. The second fits a model using untreated observations, uses it to estimate what treated observations would have looked like without treatment, and averages the resulting differences.

The main comparison excludes estimates that rely too heavily on a small number of clusters, such as states or other groups used in the statistical analysis. After this screen, there are 283 comparisons with the first method, from 28 papers, and 306 with the second, from 29 papers. Some event studies contribute separate summaries for periods before and after treatment.

How often did the estimates disagree?

The conventional and group-averaging estimates had opposite signs in 25.1 percent of their comparisons. For the imputation method, the corresponding figure was 18.0 percent. A sign reversal alone does not mean that either estimate is distinguishable from zero, or that the two estimates are statistically distinguishable from one another.

The paper also tests the difference between each pair of estimates, accounting for the fact that both are calculated from the same data. Opposite signs accompanied by a statistically significant difference occurred in 10.2 percent of group-averaging comparisons and 7.2 percent of imputation comparisons. This test does not require each estimate separately to be statistically significant.

The newer estimates were also less precise by the paper's standard-error measure: their standard errors averaged 1.61 times those of the conventional estimates for group averaging and 1.30 times for imputation. Larger standard errors indicate greater estimated uncertainty, although the averages can be influenced by a few large ratios.

An earlier economics preprint using a similar protocol found fewer sign reversals: 14.5 percent for group averaging and 10.8 percent for imputation. The business-journal sample is much smaller, however, and the paper finds that individual papers can noticeably move its summary statistics. Giving each paper equal weight, rather than letting papers with more comparisons contribute more, lowers the business-sample reversal rates to 19.9 and 16.4 percent.

Why the differences are hard to interpret

Disagreement was concentrated in unbalanced panels, where the units observed can enter or leave the data over time. In those panels, the group-averaging implementation uses whichever units are observed in each period, while the other two approaches use unit fixed effects. Differences in the populations being compared can therefore contribute to disagreement, apart from negative weights. Estimates also disagreed more in some designs without a never-treated comparison group.

The main protocol excludes specifications with time-varying controls, variables that can change over the study period. In a separate analysis of such designs, the imputation method, using the published controls, tracked the conventional estimates closely: signs differed in 3 percent of 100 screened comparisons. Group averaging differed more, but it handled the controls differently, so that comparison also reflects different choices about what to adjust for.

The available data limit the conclusions. Among 92 papers identified as having an eligible design across the three journals in the main search frame, only 41 had usable data, and 40 were re-estimated. Other packages lacked analysis data or contained synthetic or disguised data. The language-model screening and candidate search could also miss eligible studies. Finally, different estimates do not by themselves prove which method recovers the true effect: causal interpretation still depends on the design and its assumptions.

The study's practical lesson is therefore narrower than a verdict on the business literature. For the available designs examined, the choice of estimator sometimes changed even the direction of an estimate, making it useful to report a method robust to varying treatment effects alongside the conventional result.