When the risk forecast’s test needs a correction

Measuring the bad tail

Value-at-Risk identifies a threshold in a predicted distribution. For economic growth, a lower-tail threshold marks an unusually weak outcome. Expected Shortfall, or ES, goes further: it measures the average outcome below that threshold. In the paper’s application, the focus is the lowest 10% of possible growth outcomes, conditional on the predictors.

Testing these forecasts becomes difficult when models use many inputs. Regularization methods such as Lasso shrink coefficients and can exclude predictors to manage a large forecasting problem. But those estimation choices also affect the errors used in a backtest—a comparison of predictions with subsequently observed outcomes.

In Backtesting High-Dimensional Expected Shortfall, Zaichao Du, Juan Carlos Escanciano and Pei Pei propose an evaluation correction. Their arXiv preprint develops theory, simulation evidence and an application to G7 economic growth. It does not establish that the method works without its statistical assumptions.

Correcting the evaluation, not the forecast

The method has three linked stages. First, estimate the tail threshold. Second, transform observed outcomes using that threshold and estimate ES from the transformed response. This transformation reduces sensitivity to small threshold-estimation errors, but does not alone remove sensitivity to ES estimation.

The third stage supplies the additional correction. Researchers choose a test function: a predictor or transformation representing a possible model weakness. They then remove the part explained by the ES model’s regressors. The remaining component is multiplied by the transformed ES forecast error to construct the backtesting score.

For linear models, the procedure uses regularized regressions in a Double Lasso implementation. Importantly, the projection considers the full ES regressor set, not just variables selected by Lasso.

The resulting score is “orthogonal”: near the correctly specified population model, small estimation changes have no first-order effect on its expectation. This is local protection against estimation error, not immunity to incorrect models or arbitrarily poor estimates. Simply withholding evaluation observations from training does not provide the same protection.

The authors establish conditions under which the feasible test has the same large-sample distribution as an ideal test using known population quantities. Their theory covers fixed, rolling and expanding training windows. It also accounts for serial correlation rather than assuming forecast errors are independent over time.

Fewer false alarms in simulations

The main simulation comparison uses dependent predictors, heavy-tailed innovations and correctly specified forecasting models. Each configuration has 200 replications, with a nominal rejection rate of 5%.

Conventional tests rejected between 11.5% and 39.5% of the time. Orthogonal tests rejected between 3.0% and 8.5%. Both procedures used identical forecasts and initial test functions, isolating the evaluation correction in this comparison.

The improvement was not universal perfection. Separate omission experiments still showed excessive rejection under some null configurations. Detecting genuinely omitted information also depended on the chosen test function: directly testing an excluded predictor could reveal weaknesses that squared fitted ES detected poorly. Passing one diagnostic therefore does not establish overall model adequacy.

Financial information that variable selection missed

The empirical exercise forecasts year-on-year GDP growth one quarter ahead in the G7 economies. The expanded information set contains 64 nonconstant predictors, including recent growth and lagged macro-financial indicators. Models are repeatedly estimated using expanding historical samples.

Both ES specifications share a high-dimensional tail-threshold forecast. One ES model uses only recent growth; the other allows additional macro-financial predictors through Lasso.

For the United States, Lasso selected no additional financial ES predictors, making the two models’ forecasts identical. Nevertheless, the orthogonal test found evidence against excluding financial information from the growth-only ES specification. Its p-value was 0.0009, rising to 0.0120 after adjustment across 14 primary country–model tests.

That is evidence about predictive restrictions, conditional on the maintained threshold model—not evidence that financial indicators cause downturns. Nor does it automatically establish improved forecasting accuracy. A targeted extension produced lower US forecasting losses, but the authors classify that exercise as exploratory because previously inspected backtests motivated it.

What remains uncertain

The application uses historical data, not real-time release vintages, and has only 68 or 70 evaluation forecasts per country. Some fitted ES forecasts exceed their corresponding tail thresholds, violating the ordering of the population risk measures; the correction does not repair those forecasts.

The theoretical guarantees require dependence, moment, estimation-rate and nondegeneracy conditions. Empirical cross-validation does not independently verify them. Accounting for estimation can also strengthen or weaken rejection evidence, as different Canadian diagnostics illustrate.

The contribution is therefore a more careful evaluation tool, not a certificate of forecast safety: variable selection, forecast accuracy and tests of omitted information answer different questions.