Why Choosing When to Stop Can Reverse the Ranking of Experiments

The stopping rule changes the comparison

Which experiment provides better information? The answer can depend on whether its sample size is fixed in advance or adjusted as evidence arrives. In Ranking Experiments under Sequential Sampling, a theoretical preprint, Zihao Li and Tianhao Liu study decision makers who repeatedly observe one information source, choose when to stop, and then take a single action.

Their central result concerns extensive sampling: sufficiently large budgets on the expected number of observations, or sufficiently small costs per observation. Under the paper’s assumptions and a condition excluding ties, the same statistical criterion ranks experiments in both settings.

An illustrative reversal

The authors introduce a hypothetical labeling problem with two equally likely states. A decision maker must identify the true state using one of two sources. Crowd labels are correct with probability 75%. The expert source returns an informative label only 10% of the time, producing a blank otherwise; its informative labels are correct with probability 99.9%.

With exactly twenty labels, the paper reports error probabilities of about 0.9% for the crowd source and 6.1% for the expert source. Rare expert responses create a substantial chance of reaching the deadline without useful evidence.

Allowing sequential sampling reverses the comparison. With twenty labels on average, the reported error probabilities fall to approximately 0.0017% for the crowd source and 0.0005% for the expert source. These are results from the paper’s illustrative model, not measurements of actual labeling services.

The crucial difference is that an average budget is not a hard cap. A sequential rule can stop early when evidence is strong and spend more observations on inconclusive cases. The paper notes that meeting the expected budget exactly may require randomization between stopping thresholds.

Measuring evidence in every direction

The ranking criterion is directed Kullback–Leibler, or KL, divergence. For each ordered pair of possible states, it measures the average log-likelihood evidence one observation supplies in favor of the true state against the alternative. Direction matters: distinguishing one state from another can have a different evidence rate from the reverse comparison.

The authors define two forms of dominance. Large-budget dominance requires one source to deliver at least as much value in every decision problem beyond a single common expected-sample budget threshold. Small-cost dominance requires at least as much value after sampling costs when observations are sufficiently cheap, with the cost threshold allowed to vary across decision problems.

For sources whose directed KL divergences never tie, the authors prove that these rankings coincide. One source dominates another precisely when its KL divergence is strictly larger for every ordered pair of distinct states. This need not identify a winner: different directions may favor different sources.

The criterion differs from established fixed-sample comparisons. Sequential sampling can devote extra observations to ambiguous histories, making average evidence accumulation central rather than only the chance of error at a predetermined stopping date.

Reproducing information exactly

The technical foundation is an exact reproduction theorem. The authors show that a terminal signal generated by sequentially sampling one source can be reproduced using another source, with exactly the same signal distribution in every state. This concerns the final experiment, not necessarily the same observation history or stopping time.

The expected sampling requirement has a leading term determined by the largest relevant ratio of directed KL divergences. Its additional overhead is bounded by a square-root term whose coefficient depends only on the two sources, not on the stopping policy being reproduced. That uniform bound supports the common budget threshold across decision problems.

Producing different signals together

The authors also derive the limiting minimum cost per copy of producing many conditionally independent copies of a target experiment. In each state, sampling requirements are governed by the most demanding alternative-state comparison. The overall cost averages those requirements using the prior probabilities and multiplies by the observation price.

With two states, these large-scale costs are additive across conditionally independent targets. With three or more states, joint production can be cheaper when different targets have different bottleneck alternatives. Evidence unused in producing one target separately can help satisfy another target’s requirements.

Where the conclusions apply

The model assumes finitely many states and observations that are independent and identically distributed conditional on the state. Source distributions must distinguish states while assigning zero probability to the same events, and their log-likelihood ratios must have finite second moments.

The rankings are asymptotic, not guarantees for every finite budget. Ties also matter: weak KL inequalities need not establish large-budget dominance. The authors leave hard observation caps, convex sampling costs, and adaptive choice among multiple sources as questions for further research.