A Transformer’s Arithmetic Circuit Works Despite Local Geometric Distortion

Looking beyond the recovered algorithm

The preprint studies a one-layer transformer trained to add numbers modulo 113: add two inputs, then return the remainder after division by 113. Earlier research had reverse-engineered a Fourier-based algorithm in this model, showing how sine and cosine components combine to represent the sum.

That algorithm provides a mathematical reference. Multiplying complex numbers adds their phases, making complex multiplication a natural description of the Fourier computation. When one nonzero input is held fixed, ideal complex multiplication preserves local angles and stretches equally in every direction. Mathematicians call this property conformality.

But an algorithmic explanation does not establish that every internal component behaves like ideal multiplication under continuous perturbations. The authors investigate that gap rather than challenge the earlier circuit reconstruction.

Measuring an internal map

For each of five key Fourier frequencies, the researchers constructed a two-dimensional input plane from the learned embeddings. They replaced the numeric-token embeddings with their components at that frequency, while retaining the trained attention computation, positional encodings and multilayer perceptron, or MLP.

They then measured how small changes in one input affected a corresponding Fourier readout of the MLP’s contribution to the output scores. This is a restricted internal map, not the transformer’s complete input-to-output computation.

Their main instrument was the directional Beltrami coefficient. Its magnitude is zero for a conformal map. Nonzero values describe departure from that reference; values above one indicate local orientation reversal wherever the coefficient is defined. Geometrically, reversal means a small neighborhood is mapped with its orientation flipped, not that the model reverses an arithmetic answer.

These measurements depend on declared input coordinates and readout conventions. They also involve continuous interventions rather than ordinary discrete token inputs.

Correct answers, distorted local geometry

At epoch 40,000, the reconstructed model answered all 12,769 possible ordered input pairs correctly. Nevertheless, the five frequency-projected internal maps had median Beltrami magnitudes ranging from approximately 0.304 to 0.446 on the main evaluation grid.

The corresponding median ratios between strongest and weakest local stretching ranged from about 1.87 to 2.61. Infinitesimal circles were therefore generally transformed into ellipses rather than simply rotated and uniformly resized.

The authors also observed regions of local orientation reversal. Their extent differed across frequencies: one showed a narrow band around opposite-phase inputs, while another showed a substantially larger region. Stretching directions exhibited local spatial organization, rather than a single preferred direction across the entire sampled domain.

A separate check around 40 actual input-embedding pairs per frequency produced broadly comparable median magnitudes. This supports the observation beyond isolated-frequency basepoints, although the perturbation directions remained artificial interventions.

Geometry changes before the circuit metric rises

Across 500 checkpoints from one training run, the median of the five frequency-level median magnitudes fell from approximately 0.939 at initialization to 0.390 after training. Much of that aggregate reduction happened early: by epoch 1,400, the value was already about 0.391.

The measured Fourier circuit developed mainly later. The researchers tracked the fraction of centered output-score variance explained by the five key frequencies. Most of its increase followed the early geometric change, while distortion continued to fluctuate rather than decline monotonically.

This temporal ordering does not establish causation. Nor did local distortion usefully predict individual-answer margins. At epoch 9,400, correlations across 8,939 held-out pairs had absolute values no greater than 0.026. Two exceeded their individual permutation thresholds, but all remained far below the authors’ predefined meaningful-effect threshold.

What the comparisons establish

The key-frequency planes had lower median Beltrami magnitudes than most random-plane controls, although the distributions overlapped. Differences in coordinate-frame conditioning could contribute to that contrast, so the comparison does not isolate functional circuit membership as its cause.

Late-checkpoint measurements from six independent runs showed a recurring central range, but also substantial exceptions. The training-time and behavioral analyses covered only one run.

The contribution is therefore a complementary description of a known circuit: recovering its algorithm leaves measurable local geometry unexplained. The study does not show that reducing distortion improves computation or that these findings extend to larger models.