How Much Can an Economic Study’s Assumptions Fail Before Its Conclusion Changes?

A larger estimated effect does not necessarily provide more robust evidence. That is a central lesson of Filippo Palomba’s preprint, Sensitivity Analysis in Population Shares, which develops a common framework for asking how far economic conclusions can move when distributional assumptions are relaxed.

The Princeton University researcher applies the method to experimental dropout, unobserved preferences in a marriage model and assumptions about the timing of Chinese railway construction. The contribution is methodological: it measures departures from assumptions in interpretable shares, while distinguishing uncertainty about the model from uncertainty caused by limited data.

A share, not a verdict

Researchers often need assumptions about distributions they cannot fully observe. An experiment may assume that people who complete a follow-up survey are comparable to those who do not. A structural model may impose a particular distribution on hidden preferences.

Palomba introduces a sensitivity parameter between zero and one. In one formulation, a share of the population follows an unrestricted alternative distribution, while the remainder follows the researcher’s baseline. In another, suited to experimental attrition, the method limits differences between two groups’ outcome distributions while preserving their observed mixture.

For each permitted share, the framework calculates the lowest and highest values consistent with the remaining restrictions. These are sensitivity bounds, not estimates of how many people actually violate an assumption. Allowing a conclusion to change does not establish that it is wrong.

At zero, the exercise recovers the baseline result when the baseline identifies a unique value. At one, the selected distributional restriction disappears. Palomba shows that the resulting bounds are the tightest permitted by the maintained support, moment and, where applicable, mixture restrictions—not bounds free of all assumptions.

Using convex analysis, he converts optimization over distributions into finite-dimensional inner problems. Structural applications can still require an additional search over model parameters. The paper also develops estimators and a specialized bootstrap procedure for statistical inference.

Tutoring: the smaller estimate is more robust

Palomba revisits Michela Carlana and Eliana La Ferrara’s online tutoring experiment among disadvantaged Italian middle school students. Its two waves took place during school closures in 2020 and regular in-person schooling in 2022.

Among respondents, the estimated mathematics effect was larger in 2020: 0.207 standard deviations, compared with 0.176 in 2022. But response rates differed much more sharply between treatment and control in the earlier wave: 87.9% versus 46.3%, against 92.3% versus 72.3% in 2022.

Without covariates, Palomba estimates that the positive sign ceases to be guaranteed when the share allowed to depart from the random-attrition benchmark reaches roughly 10% in 2020 and 25% in 2022. Accounting for sampling uncertainty reduces those thresholds to about 1.7% and 5.1%, respectively, at the paper’s 5% significance level.

These calculations concern students who would respond under either treatment assignment, not necessarily all students enrolled. They also maintain the assumption that tutoring cannot reduce an individual’s likelihood of responding. The later wave is therefore more robust within this specified exercise, despite its smaller starting estimate.

Other assumptions, similar questions

In a marriage-matching application, Palomba examines a model-based difference in marital utility between college-graduate women and women with some college. The baseline assumes a Gumbel distribution for hidden preference shocks.

For the selected cohort, the baseline premium is negative. The upper sensitivity bound remains negative when 10% of women may draw non-Gumbel shocks, but becomes positive at 20%. This concerns a model-based utility comparison, not a wage return to college.

A third application revisits Kirill Borusyak and Peter Hull’s estimates of employment effects from improved market access through Chinese high-speed rail. Palomba finds that allowing an alternative assignment mechanism for roughly 19% of shock draws makes the sensitivity bounds include zero, both with and without geographic controls. Here the realized sample stays fixed: the bounds describe changes in the estimate, without incorporating sampling uncertainty.

What more data cannot fix

Palomba calls the first share at which a conclusion can fail its “breakdown budget.” A “certified” counterpart also accounts for sampling uncertainty. Their gap helps distinguish fragility that more data could reduce from fragility that persists even with a very large sample.

The guarantees are conditional. The general inference theory requires technical regularity conditions, including compact support and continuity; an attrition-specific result relaxes compactness. The paper explicitly leaves a general relaxation for the unbounded shocks in the marriage application to future work. General inference also holds some auxiliary parameters fixed, although the attrition analysis separately accounts for its estimated mixing share.

The preprint thus offers a structured comparison of conclusions under chosen departures—not a test proving assumptions correct, or a universal ranking independent of the restrictions retained.