When a few observations carry a regression, loops offer another way to test it

Fixed-effect regressions help researchers compare observations while accounting for persistent differences between groups. In investment data, for example, firm effects absorb stable differences between companies, while year effects absorb changes shared across companies. But after these adjustments, the variation available to estimate a particular coefficient can be surprisingly concentrated.

In a preprint, Stanisław M. S. Halkiewicz develops a graph-based implementation of an exact testing method for this situation. The contribution is not the invention of sign-flipping: the paper explicitly builds on Guillaume A. Pouliot’s exact t-test. Its focus is how fixed-effect designs determine which comparisons can support such a test, and how much useful variation those comparisons retain.

Many rows, few influential contributions

The relevant quantity is not simply the number of observations. It is how the explanatory variable varies after the other regressors have been removed. If a few observations carry much of that remaining variation, the usual justification for a bell-shaped reference distribution can fail.

Unequal error variances can make the problem worse. An observation with substantial identifying variation and unusually noisy outcomes may dominate the statistical evidence. The paper distinguishes concentration visible from the explanatory variables alone from a realized diagnostic that also uses fitted residuals. The latter is a warning signal, not generally a reliable estimate of the underlying population concentration.

In one calibrated simulation, a conventional degrees-of-freedom-corrected t-test intended to reject a true null hypothesis only 5% of the time instead rejected it 58.5% of the time. A leverage-adjusted HC2 test rejected 33.4% of the time. These are results for a particular concentrated design and symmetric error process—not general failure rates for either procedure.

Cancel first, then flip signs

The alternative starts by constructing contrasts: weighted combinations of observations chosen using the explanatory variables and group structure, without consulting outcomes. Their weights cancel the fixed effects exactly. Additional nuisance regressors must also be canceled.

Under a proposed coefficient value, each contrast then contains only a combination of errors if the null hypothesis is true. When these combinations are independent and symmetric about zero, reversing their signs preserves their joint distribution. Comparing the observed statistic with statistics generated by those sign reversals supplies a valid p-value.

“Exact” here means finite-sample false-positive control: rejection probability is no greater than the chosen level under the assumptions. It need not equal that level, because the available sign patterns are discrete. Error variances may differ arbitrarily. Dependence is allowed within independent, centrally symmetric blocks, provided the selected supports do not split those blocks.

For two-way fixed effects, the construction has a useful geometric interpretation. Represent workers and firms, for example, as two sets of network nodes, with observations joining them. Alternating positive and negative weights around a closed loop cancel both sets of fixed effects. These cycle contrasts span the space containing the remaining explanatory-variable variation.

Retaining information costs usable signs

The paper proposes structured ways to select cycles without reusing observations. Its capture ratio measures the fraction of identifying variation retained by the selected contrasts. A higher ratio is desirable, but not sufficient: merging supports can retain more variation while leaving fewer independently flippable signs.

That trade-off can prevent rejection altogether. With only three active supports, the default two-sided, fully enumerated test cannot produce a p-value below 0.25. More elaborate packing cannot make three signs behave like many independent comparisons.

The capture ratio also has an efficiency interpretation under additional conditions, including equal error variances and sufficiently dispersed contributions. That interpretation does not automatically extend to concentrated or arbitrarily heteroskedastic settings.

An investment example—and its limits

The empirical illustration uses 220 observations from 11 firms in the Grunfeld investment dataset. The regression relates investment to capital, controlling for market value, firm effects and year effects. One observation accounts for 73.9% of the realized squared score contributions.

Because ordinary four-observation cycles do not cancel market value, the method instead uses supports built from two firms and three periods. Thirty-two selected contrasts retain 62.7% of identifying variation. Inverting the sign-flip test using 99,999 sampled flips gives a reported 95% confidence interval of approximately 0.150 to 0.450 for the capital coefficient. This is a conditional association, not a causal effect.

The crucial qualification remains symmetry. The application does not establish that assumption empirically. The paper also proves large-sample validity without symmetry when independent, centered contrast contributions satisfy a fourth-moment negligibility condition, but a persistently dominant contribution is not covered by that result. The preprint therefore offers a conditional remedy for concentrated evidence—not assumption-free inference.