Better Bayesian sampling by jumping between equivalent explanations

A valid distribution is not enough

Bayesian analysis combines a likelihood, describing how a model generates observations, with a prior distribution over its parameters. The resulting posterior represents uncertainty after observing the data. But having a mathematically valid posterior does not mean a computer has explored it reliably.

In an arXiv preprint, Toru Kitagawa and Yizhou Kuang propose turning a source of statistical ambiguity into a computational advantage. Their identification-aware Markov chain Monte Carlo, or MCMC, supplements conventional sampling with jumps between parameter settings that imply the same distribution of observations.

The distinction matters in economics: different structural explanations can fit the data identically while implying different responses to economic shocks. A sampler that visits only some of those explanations can misrepresent uncertainty without an obvious deterioration in fit.

What makes a jump legitimate?

MCMC generates a sequence of dependent parameter draws designed to represent a target distribution. Local updates can explore one region effectively yet struggle to cross between separated regions or travel along extended directions where the likelihood is unchanged.

The authors use observational equivalence to construct additional moves. Two parameter settings are observationally equivalent when they imply the same observable-data distribution for every sample size—not merely when they happen to fit the observed sample similarly.

A simple example involves observations whose mean is the sum of two unknown components. Increasing one component while decreasing the other by the same amount leaves the observable distribution unchanged. Instead of gradually traversing these alternative decompositions, a sampler can draw directly along their equivalence set.

The proposed hybrid randomly chooses between an ordinary local update and an equivalence-based “teleport” update. The latter must preserve the posterior distribution conditional on the chosen equivalence set. When direct conditional sampling is unavailable, the paper supplies corrected proposal mechanisms.

Crucially, equal likelihood does not mean equal posterior weight. Priors can favor some equivalent explanations over others, and transformations between parameter representations can change the relevant probability measure. Teleport moves must account for both effects rather than jump uniformly by default.

When the mathematics promises improvement

The paper studies separated posterior modes and extended likelihood-flat sets. Its main theoretical comparisons concern random-walk Metropolis sampling.

The authors measure performance through a spectral gap, a mathematical quantity that controls worst-case asymptotic variance for posterior averages under the stated reversible-chain conditions. Their bounds show how local sampling can deteriorate as modes move farther apart or flat sets expand, while identification-aware sampling retains a positive gap under specified assumptions.

These are conditional results, not a universal guarantee. They require adequate local exploration and communication supplied by the teleport moves. The curved-set result also imposes geometric and probability bounds that must remain uniform as the problem changes. Extensions to Hamiltonian Monte Carlo and the No-U-Turn Sampler, or NUTS, require additional conditions.

Simulations illustrate the mechanisms using interchangeable labels in Gaussian mixtures and alternative decompositions of a Gaussian mean. Relabeling helps explore label-dependent quantities; quantities already invariant to labels do not change simply because labels are swapped.

Testing structural uncertainty in economics

The empirical exercises use structural vector moving-average models, which describe how shocks affect economic variables over time. Equivalent representations preserve the observable distribution but can assign different responses to structural shocks.

The larger application has 153 free parameters and uses quarterly U.S. productivity growth, GDP growth and a real interest rate from 1954Q3 to 2007Q4. It compares NUTS with identification-aware alternatives under informative and bounded-uniform priors. The larger model’s teleport proposals use rotations only, not every transformation generating observational equivalence.

Under the bounded-uniform prior, identification-aware NUTS reduced the mean scaled coefficient discrepancy from a numerical conditional-prior reference from 0.125 to 0.033—approximately 74 percent. Teleport updates accounted for 2.1 percent of transition time. Under the informative prior, the corresponding discrepancies were already similar: 0.027 and 0.026.

Better exploration is not proof of convergence

The references are not independently known full posteriors. Each averages conditional completions over conditioning values visited by its own sampler. Improved agreement therefore diagnoses aspects of exploration, not complete convergence.

Nor did every diagnostic improve: median autocorrelation-based effective sample size was lower for identification-aware NUTS in the larger bounded-uniform exercise, despite better coefficient calibration.

The preprint’s practical message is that weakening a prior can expose equivalent explanations that local sampling misses. Apparent stability across priors may then reflect incomplete computation rather than genuinely stable conclusions. Exploiting known equivalences can help reveal that uncertainty without resolving the underlying identification problem.