Balancing statistical tests when no single choice is best

A statistical test can be good at detecting one possible effect and less effective at detecting another. A new preprint asks how to balance those competing strengths without losing control of false positives—and provides an algorithm with explicit approximation guarantees in a specific Gaussian setting.

Choosing which shortfalls to minimize

In Approximate minimax regret for conditionally similar tests, Andrés Aradillas Fernández of MIT studies how to choose among tests that share the same false-positive protection. Version 1 was posted to arXiv on November 4, 2026; the work is a preprint.

The setting is motivated by instrumental-variables analysis, including situations where instruments may be weak. The paper starts from a problem: no single test necessarily has the highest probability of detecting an effect at every alternative to the null hypothesis.

That detection probability is called power. Rather than maximizing average power using permanently fixed weights, the proposed method minimizes the largest power shortfall across a chosen collection of alternatives. The paper calls this shortfall regret.

Its benchmark is demanding but specific. At each alternative, regret compares a test’s power with the best power attainable there by any test in the same conditionally similar class. Different alternatives can have different best tests; the algorithm seeks a compromise that keeps the worst shortfall small.

False-positive control built into each step

The mathematical experiment involves two jointly Gaussian estimates: a reduced-form estimate and an estimate of instrument strength. Their covariance matrix is assumed known, positive definite and independent of the unknown parameters. The hypothesis concerns a scalar effect, while instrument strength is an unrestricted nuisance parameter.

A transformation produces two independent normal variables, called S and U. Under the null hypothesis, the conditional distribution of S given U does not depend on the nuisance parameter.

This allows the author to impose conditional similarity: for almost every value of U, a test must reject the null with probability exactly equal to the chosen significance level. That also gives the same unconditional rejection probability for every nuisance-parameter value under the null.

The distinction matters computationally. The paper’s approach builds this restriction into every test it constructs, allowing its iterative search to focus on alternatives where power regret is high rather than adjusting null-distribution weights to enforce false-positive control.

Updating weights, then averaging tests

The algorithm begins with equal weights on a finite grid of alternatives. At each iteration, it constructs a test maximizing weighted average power while satisfying the conditional restriction. A conditional Neyman–Pearson calculation supplies this best response.

It then evaluates or simulates the test’s power at each alternative. An exponential-weight update increases emphasis on alternatives with greater regret. This is a form of stochastic mirror descent: the weights change repeatedly as the procedure searches for a better balance.

The returned result is the average of the tests, not a new test constructed from average weights. Operationally, one can select an iteration uniformly at random and apply that iteration’s test. The paper warns that constructing a best response to average weights can underperform.

The author proves finite-iteration bounds for both exact and simulated power calculations. With exact powers and a specified step size, the excess worst-case regret bound decreases proportionally to the square root of the logarithm of the grid size divided by the iteration count. Simulated powers add a probability-qualified error term.

Crucially, simulation affects the regret guarantee, not conditional size—provided each conditional best response is computed exactly. Averaging preserves that size restriction even after conditioning on the simulation history.

What the numerical example shows

The illustration uses a 5% significance level and just two alternatives, deliberately chosen to expose a trade-off between negative and positive effects. Their pointwise power benchmarks are 0.4424 and 0.6388.

An equal-tail test has powers of about 0.323 and 0.516, with maximum regret of 0.1228. The initial equally weighted conditional best response improves power at both alternatives and reduces maximum regret to 0.0819.

After 4,251 iterations, the averaged test has powers of about 0.365 and 0.563, and maximum regret of 0.0776. The later improvement therefore comes from a trade-off: more power at the first alternative, less at the second.

For this run, the theoretical settings guarantee at least 95% probability of being within 0.02 of the optimal grid regret, assuming exact conditional best responses. That does not mean regret itself is below 0.02. A numerical lower bound leaves a reported optimality gap of about 0.00065.

Where the result stops

These are guarantees within the full conditionally similar class in this Gaussian experiment—not among all level-controlled tests. They cover the specified finite grid, not every possible alternative. The numerical results come from one simulation run, and regret need not improve monotonically with iteration.

The author presents extensions beyond the Gaussian setup as future work. The demonstrated contribution is an approximation method that balances competing power losses while separating that optimization from conditional false-positive control.