Comparing banks’ risk forecasts without fixing a stopping date

A comparison, not a certificate of safety

Banks forecast how much they might lose. Backtesting checks those forecasts against subsequent outcomes. But a regulator may need something more specific: evidence that a bank’s internal model performs better or worse than a prescribed benchmark.

In the preprint Comparative e-backtesting, Zhanyi Jiao, Qiuqi Wang and Yimiao Zhao develop a framework for making that comparison repeatedly as observations arrive. The goal is regulatory assessment, not simply choosing a winner from a collection of models.

The framework covers Value-at-Risk, a loss threshold associated with a specified probability, and expected shortfall, which describes losses in the tail beyond that threshold. Expected shortfall is evaluated jointly with Value-at-Risk. The authors also examine expectiles, forecasts defined by giving unequal weight to squared errors on either side of the prediction.

Keeping repeated checks statistically valid

Repeatedly inspecting a conventional fixed-date significance test can undermine its advertised error rate. The proposed method instead uses e-processes: nonnegative measures of evidence constructed to support monitoring and stopping at data-dependent dates.

The comparison starts with a scoring function appropriate to the chosen risk measure. Lower expected scores indicate better forecasts. The quantity of interest is the average difference between the internal model’s and benchmark’s conditional expected scores over an assessment period—not merely the difference between their observed scores.

To estimate that quantity, the researchers specify a family of conditional probability distributions for losses, indexed by unknown parameters. They compare each candidate distribution with an auxiliary predictive density fitted using information available before the next observation. These likelihood comparisons build e-processes and a sequence of confidence regions for the unknown parameters.

The procedure then calculates the smallest and largest average expected score differences compatible with the retained parameters. This permits comparisons even when score differences have no fixed finite bound, an obstacle for some existing sequential methods.

Three colors, two reporting schemes

The authors translate those bounds into three outcomes. Red means the entire range indicates worse average performance by the internal model. Green means it indicates better performance. Yellow means neither direction is established.

Yellow does not demonstrate equal performance, and green does not certify that either model correctly forecasts risk. Both conclusions concern relative performance under the selected scoring function.

An overall assessment compares forecasts from a chosen starting point through the current reporting date. A resetting assessment closes an interval once a red or green conclusion appears, then begins another. Crucially, resetting the assessment period does not discard the history used to estimate forecasts or construct the likelihood confidence sequence.

Under the paper’s conditional-model assumptions, the exact procedure bounds the probability of ever making a wrong directional report across assessment periods by the chosen error level. That also supports its false-discovery-rate guarantee for resetting reports. It is not a blanket guarantee across an arbitrary collection of separately tested model pairs.

What the examples show

In a controlled experiment, the authors generated normally distributed losses and compared several Value-at-Risk reporting strategies over 1,000 simulation runs. Against a perfect-information benchmark, forecasts that underreported the true Value-at-Risk by 20%, with added noise, received red terminal decisions in 85.5% of runs. The corresponding rate for 20% overreporting was 51.9%.

Yet those two imperfect strategies were distinguished from one another in only 0.3% of runs. Detecting that both forecasts are inferior to an excellent benchmark can therefore be much easier than ranking them against each other.

The empirical analysis used NASDAQ Composite losses derived from returns between January 2003 and May 2025. Following estimation and initialization, assessment began in December 2006. Six forecasting approaches combined different distributional assumptions with parametric, historical-residual or extreme-value methods.

At the final date, the procedure resolved seven of 15 distinct model pairs for Value-at-Risk, five for expectiles and six for the joint Value-at-Risk–expected-shortfall assessment. Different comparisons favored different methods; the results did not establish a universally best model. Resetting assessments also produced opposite directions over different intervals for some pairs.

Where the guarantees stop

The central limitation is model dependence. The theoretical guarantees assume that the true conditional loss distribution belongs to the specified parametric family. The auxiliary predictor can be misspecified without invalidating those guarantees, but that flexibility does not extend to the underlying loss-model family. Auxiliary prediction quality also affects how readily the test reaches conclusions.

Implementation adds another qualification. For the time-series examples, the authors restrict parameter searches to a numerical box and use local optimization to approximate the score bounds. Those computations do not certify the exact bounds required by the theorem.

The preprint thus offers a framework for continuously monitored, benchmark-relative assessment—not an assumption-free safeguard against financial risk. Its practical reliability depends on both the loss-model assumptions and the numerical bounds used to reach each colored conclusion.