A more informed way to train neural networks without backpropagation

Why random search struggles

Backpropagation trains neural networks by calculating how each parameter contributes to prediction error. It requires a backward computation through the model, using stored or reconstructed intermediate information. That can be awkward for physical, neuromorphic or asynchronous systems whose operations do not naturally support such calculations.

Zeroth-order optimisation offers another route: make small changes to a network, run it forward, and observe whether its loss—the numerical measure of prediction error—changes. Those observations provide estimates of useful learning directions without computing derivatives through the network.

The difficulty is scale. With many parameters, random changes tend to point poorly toward the true gradient. The paper’s analysis shows that, for standard isotropic weight perturbations in a local linear approximation, estimation error grows roughly with the number of parameters divided by the number of probes. A probe is a perturbed forward evaluation and its corresponding loss observation.

Justus Hübotter, Serge Thill, Marcel van Gerven and Nasir Ahmad, at Radboud University’s Donders Institute, propose making these probes less wasteful. Their paper, An Educated Guess, was submitted to arXiv on 5 November 2026 as version one. It is a preprint; the supplied record does not establish peer-reviewed publication.

Using the forward pass as a guide

The central observation is that a layer’s input already reveals which weight changes can affect its output on the current data. Changes outside that input-supported space spend perturbation effort without changing the layer’s response.

The researchers collect layer inputs into a matrix and use its uncentred second moment to describe their geometry. For convolutional layers, those inputs are the image patches processed by a shared kernel. This matrix identifies feature directions and their relative strength.

Their Spectral Zeroth-Order family, SZO-κ, shapes random weight changes according to that geometry. The parameter κ controls how strongly the search favours dominant feature directions. At κ = 0, it recovers ordinary weight perturbation. SZO-1 distributes exploration across feature-supported directions, while the infinite-κ limit concentrates on the leading direction.

This is an intentional bias–variance trade-off. Concentrating the search can reduce noise, but may neglect weaker directions that matter for learning. The derivation assumes that the unknown downstream error has no preferred direction and that perturbations are small enough for a linear approximation. The authors acknowledge that training can violate the first assumption, making the objective a heuristic rather than an exact optimum.

For fully connected layers with individual-example feedback, all positive-κ variants reduce to node perturbation, which perturbs activities rather than searching the entire weight space. That equivalence does not hold for convolutions, where shared weights couple responses across spatial positions.

What the experiments show

The team trained a two-layer fully connected network on Fashion-MNIST and a six-layer convolutional network on CIFAR-10. Main experiments used stochastic gradient descent for 250 epochs, with learning rates selected on a held-out validation split. Final SGD evaluations used ten random seeds; gradient-alignment measurements used twenty initialisations.

The feature-guided estimates aligned better with backpropagation gradients than ordinary weight perturbation. On Fashion-MNIST, SZO and node perturbation trained identically under individual-example feedback, as predicted. On CIFAR-10, SZO-1 performed best among the perturbation methods with SGD and Adam. With the SOAP optimiser, however, node perturbation and the hybrid Node-Through-Weight Perturbation method achieved the highest perturbation-method accuracies.

Backpropagation remained best on both datasets. Its SGD test accuracy reached 89.9% on Fashion-MNIST and 83.6% on CIFAR-10. The work therefore demonstrates improvements within perturbation-based learning, not a replacement that matches standard training.

Comparisons also involve unequal feedback. Most methods received a separate loss difference for each example, while ordinary weight perturbation received one batch-wide difference. At the default batch size of 128, that means 128 observations versus one per probe. The paper explicitly distinguishes these regimes.

Costs and boundaries

Main experiments perturbed one layer at a time to reduce interference between layers, requiring extra forward computation. Perturbing all layers simultaneously reduced compute but lowered CIFAR-10 validation accuracy: SZO-1 reached about 35%, versus 46% with layer-wise probing after 250 epochs.

Spectral shaping can itself be expensive. The authors provide implementations that avoid full eigendecompositions for useful variants; their convolutional infinite-κ implementation uses a representative-patch approximation rather than the exact leading eigenvector.

On an NVIDIA A100, their research implementation of SZO-1 took 3.9 times backpropagation’s time per CIFAR-10 training step, while using less peak allocated memory: 388 versus 573 MiB. No neuromorphic hardware demonstration is reported. The contribution is a grounded way to guide forward-only learning—not evidence yet of faster training or successful deployment on unconventional devices.