How normalization can hide extreme sensitivity in deep neural networks

Two meanings of stability

A neural network can keep its internal activations bounded while remaining extremely sensitive to small changes. That distinction anchors this study of normalization, focusing on batch normalization.

The researchers examine two Lipschitz constants: quantities that bound how much an output can change relative to a perturbation. Input Lipschitzness measures sensitivity to changes in the data while weights stay fixed. Parameter Lipschitzness measures sensitivity to changes in the weights while the input stays fixed. These quantities enter mathematical guarantees about robustness, generalization and optimization.

The authors construct normalized networks in which both constants grow exponentially with depth. In fixed-width constructions, that also means exponential growth with the number of weight parameters. This is an existence result: such instability can occur, not a claim that every normalized network exhibits it.

A controlled construction

The core theoretical model alternates matrix multiplication with column-wise normalization, rescaling each nonzero column to unit length. It initially uses linear activations, removing activation nonlinearities as a possible explanation for amplification.

For the input-sensitivity result, every weight matrix is orthogonal. These transformations preserve distances before normalization, and the normalized activations maintain a fixed overall norm. Nevertheless, the complete network can separate two initially close inputs dramatically. Controlling the weight matrices alone therefore does not control the sensitivity of the composed network.

The central example compares a rank-deficient matrix with a nearby full-rank matrix. Informally, rank counts independent directions in a matrix. The full-rank input contains a tiny additional direction that the other input lacks. Carefully selected transformations and normalization progressively make the full-rank representation better conditioned, while the rank-deficient representation cannot gain the missing direction.

The final representations remain a substantial distance apart even though the initial difference becomes exponentially small as depth increases. The researchers transfer this construction to weight sensitivity by encoding the two input matrices into alternative first-layer weights and keeping subsequent layers unchanged.

From output separation to wrong predictions

Large output differences do not automatically imply classification errors. To establish that connection, the authors append a deliberately chosen binary classifier to their constructed network.

Its decision boundary separates the two final representations. The clean input has a positive score margin, yet an exponentially small, rank-creating perturbation flips its predicted label. This demonstrates an actual classification failure within the construction, rather than merely a pessimistic stability bound. It does not establish that an arbitrary trained classifier has the same boundary or vulnerability.

Testing perturbation geometry

The researchers also investigate whether this geometric distinction appears in image models. They use batch-normalized convolutional networks trained on MNIST and Fashion-MNIST, plus a pretrained ResNet56 on CIFAR-10.

Their comparisons include independent Gaussian pixel noise, structured row and column offsets, and perturbations aligned with an image's singular directions. Singular-value perturbations change the strengths of existing components rather than introducing new directions before clipping. Perturbations are rescaled to match their energy before pixel values are clipped to the valid range.

The reported results show that same-subspace perturbations generally preserve accuracy, confidence and classification margins better, with fewer prediction changes. Pixel noise strongly increases several effective-rank measures. Row and column offsets can also damage predictions despite producing much smaller changes in those rank measures, underscoring that rank alone is not a complete predictor.

A separate two-dimensional experiment tests the theoretical construction using a greedy search over rotations. Constructed input pairs show exponential sensitivity, while the median and 95th percentile for generic random local pairs remain nearly flat. That contrast is important evidence that the result concerns a structured worst case, not typical perturbations everywhere.

What the guarantees do—and do not—say

For optimization, the paper shows how guarantees that depend polynomially on an objective's Lipschitz constant can become exponential in parameter count for the constructed networks. This weakens those guarantees; it does not prove that practical training must take exponential time. Likewise, an uninformative generalization or robustness bound is not itself evidence of poor test accuracy.

The authors propose probabilistic Goldstein stationarity, a weaker optimization target that limits the fraction of directions with a large directional margin. Their function-value algorithm avoids polynomial dependence on the Lipschitz constant, but achieves this relaxed condition rather than the original worst-case requirement.

Important limits

The main proofs use idealized normalization without mean subtraction; appendices extend the construction to centered normalization and specially configured ReLU networks. The authors also discuss regularization near zero columns. These results should not be read as a uniform quantitative prediction for every practical normalization configuration.

Matching noise energy before clipping does not guarantee equal energy afterward, and clipping can change subspace structure. The image experiments support the proposed mechanism without proving it explains every observed failure. The central lesson is narrower: bounded activations and controlled weights can coexist with severe, geometry-dependent worst-case sensitivity.