Shuffling variable order can make statistical model checks more reliable

A statistical model can describe each variable reasonably well yet misrepresent how they move together. Checking such a model requires examining the joint observation: for example, one day’s returns across nine currencies. In the preprint Calibrated Order-Randomized Rosenblatt Tests, Mehrdad Pournaderi investigates a complication in a classical checking method: its ability to detect a mismatch can depend on the order in which variables are processed.

The study, whose second arXiv version was posted on November 6, proposes repeating the calculation under random orderings and combining the evidence. Its simulations support that approach in selected Gaussian settings, not as a universal improvement for every distribution or kind of model failure.

One observation, several views

The Rosenblatt transform processes variables sequentially. For each variable, it asks where the observed value lies within the distribution predicted after accounting for the preceding variables. If a fully specified, continuous joint model is correct, the resulting positions are independent uniform values between zero and one.

That turns a complicated model check into a simpler statistical test. But changing the sequence changes which variables predict which others. A mismatch may appear as one conspicuous residual in one ordering and several less conspicuous residuals in another. Each hypothesis here concerns a single multivariate observation, not a sample of repeated observations.

For Gaussian mean shifts, the paper shows that reordering preserves the total signal strength measured by Mahalanobis distance, which accounts for correlations. It changes the signal’s distribution across transformed coordinates rather than creating information. The tests examine both positive and negative extremes, because changing the order can move a discrepancy between tails.

The proposed procedure draws multiple orderings independently of the data, scores each view and combines them using rules that allow dependence. This matters because the views are recalculations of the same observation, not independent evidence.

One combination averages e-values: nonnegative evidence statistics whose expectation under the null hypothesis is at most one. Their average remains an e-value even when the components are dependent. The paper also examines two ways to combine p-values.

Calibration makes the comparison meaningful

Validity and useful sensitivity are separate issues. The nominal combination rules can be conservative, rejecting far less often than their allowed false-alarm rate. Conversely, simply inserting an estimated covariance matrix into formulas intended for a known Gaussian distribution can produce excessive false alarms.

For an unknown Gaussian population, the study uses a classical conditional probability transformation that accounts for parameter estimation and gives each ordering valid scores. A parametric bootstrap then reduces the pooled tests’ conservativeness: it simulates training data and test observations from the fitted model, re-estimates parameters and repeats the calculation.

For pooled statistics under estimated parameters, this bootstrap is approximate, not a finite-sample guarantee. Across a robustness sweep, calibrated single-test false-alarm rates ranged from 4.2% to 6.0% around a 5% target. In one multiple-testing experiment with limited training data, the p-value-averaging method produced a false discovery rate of 15.2% against a 10% target.

Gains depend on the departure

The simulations primarily studied Gaussian mean shifts under a covariance structure with equal pairwise correlations. Departures were compared at equal total signal strength, including shifts concentrated in one coordinate and shifts spread across all coordinates.

In one ten-dimensional experiment with known parameters, the calibrated e-value average detected 72.8% of concentrated departures and 66.2% of diffuse departures. A symmetric-root benchmark, which avoids ordering, detected 72.3% and 37.6%, respectively. The chi-squared energy benchmark detected 62.9% regardless of departure shape.

These results illustrate robustness across the tested shapes, not universal superiority. The paper’s theorem establishes an advantage in expected logarithmic evidence from order averaging; it does not guarantee higher detection probability at a fixed threshold. Twelve orderings captured most of the averaging benefit in the examined designs, but some individual decisions still changed with the random seed.

Currency-market checks—and limits

The application fitted a Gaussian model to the preceding year’s returns for nine currencies against the US dollar, then tested days in 2016–2025. Screening 2,497 days with pooled e-values flagged 78 under the issued-forecast interpretation, or 62 under a different hypothesis allowing for uncertainty about a shared Gaussian population. Prominent flagged episodes included Brexit and the COVID market disruption.

However, pooling’s day-level rejection-rate gain over one random ordering was statistically indistinguishable from zero. Without ground-truth labels, those flag rates do not measure detection power or establish the causes of unusual days.

Broader covariance structures, heavier-tailed models and departures involving scale or dependence remain insufficiently explored. The base tests can also miss reduced variability. This report uses the supplied PDF text extraction, which may omit figures and damage mathematical layout. The work remains a preprint.