AI’s Algorithmic Efficiency Gains Depend on How Much Compute You Have

Training a language model takes computing power, often measured in floating-point operations, or FLOPs. Algorithmic improvements can help a model reach a given performance using fewer FLOPs. A widely cited estimate put the increase in language-model pretraining efficiency from 2012 to 2023 at about 22,000-fold. Hans Gundlach and colleagues ask which changes in training methods could account for such gains—and whether those changes help equally at every scale.

Their paper argues that efficiency gains depend on compute scale. It combines controlled experiments, scaling-law analysis and estimates from earlier research. Its central message is not that algorithms do not matter, but that one efficiency multiplier can conceal important differences. The manuscript identifies itself as a NeurIPS 2026 contribution and is available as an arXiv preprint; the claims described here should be read as research findings, not as a settled accounting of AI progress.

Testing changes one at a time—and together

The researchers first ran ablation experiments: they changed or removed training components and measured how much compute was needed to reach a shared validation-loss threshold. Components included activation functions, positional encodings, normalization methods, learning-rate schedules and optimizers. These tests used small Transformer models. The team also combined changes to test whether their gains simply multiplied.

Most tested components made modest differences in these experiments. All the tested innovations together produced a reported 7.9-fold compute-equivalent gain relative to the study’s LSTM baseline. By comparison, multiplying the estimated gains from individual changes would predict 21.7-fold. The smaller combined result suggests that improvements can overlap or interact; adding isolated estimates may overstate the benefit of combining them. The post-Transformer changes alone had a separate measured combined gain of 1.33-fold.

Next, the researchers trained models at different sizes to compare scaling patterns. They examined a modern Transformer, a Transformer with several later components reverted, and an LSTM intended to represent pre-Transformer language modeling. The models used the same dataset and tokenizer, with learning-rate tuning across sizes. The team fitted scaling relationships to the training results and compared how much compute each approach would need for similar performance.

A possible scale-dependent advantage

Within this experimental setup, Transformers improved more rapidly with increasing compute than the LSTM baseline. The authors interpret the difference in fitted scaling exponents as evidence that the Transformer transition brought scale-dependent gains: its advantage grew at higher compute levels. By contrast, many of the individual Transformer components they tested showed little scaling difference.

The paper also analyzes the shift from Kaplan-style to Chinchilla-style allocation of model parameters and training data. Under the authors’ assumptions, this shift offers another scale-dependent source of efficiency. Combining experimental results with extrapolations and estimates from the literature, they project about 3,360-fold efficiency gains relative to an LSTM at 2023 compute levels. In that reconstruction, the LSTM-to-Transformer transition and the scaling-policy shift account for 88% of the log-scale gains. This percentage describes the authors’ extrapolated decomposition, not a direct measurement of every historical training run.

At low compute, the experimentally measured changes and selected literature estimates add up to less than 100-fold, far below the cited 22,000-fold historical estimate. The authors argue that scale-dependent innovations could explain part of the gap: a benefit that is modest in small experiments may become much larger when extrapolated to high compute. Their reconstruction still falls short of the historical estimate, leaving room for factors not captured, such as data selection and tokenizer changes.

The yardstick matters

A compute-equivalent gain compares training approaches at a specified performance level. If an approach’s advantage changes as compute grows, the reported multiplier depends on both the compute level and the reference approach.

For example, the same sequence of mixture-of-experts Transformers can appear to become increasingly efficient when compared with LSTMs, yet show a roughly constant advantage when compared with dense Transformers. The models being assessed are unchanged; the benchmark differs. The authors therefore argue that efficiency claims should name the reference algorithm and compute scale.

What the evidence cannot settle

The study does not experimentally cover every important development. The authors did not run dataset ablations, and estimates for some advances, including mixture-of-experts architectures, come from literature rather than matched tests. Their scaling experiments are small compared with frontier training runs. Extending fitted relationships to much larger compute budgets is an extrapolation that depends on assumptions, especially the difference between LSTM and Transformer scaling exponents. The paper’s sensitivity analysis shows that alternative assumptions can substantially change projected totals.

Results also depend on choices such as hyperparameter tuning and the LSTM baseline; better tuning could alter the comparison. And the study measures pretraining FLOP efficiency, not every kind of progress. Some methods may improve stability, wall-clock time, energy use or model usefulness without reducing FLOPs in this measure.

The takeaway is about interpreting efficiency estimates, not a definitive tally of AI progress. If gains vary with scale, small experiments may miss valuable methods, while comparisons anchored to older baselines may not describe progress relative to current systems. Larger, carefully matched studies are needed to test how broadly the proposed explanation applies.