Could a language model’s attention work like a radio?

The question: what is attention doing?

In a transformer, attention helps each token gather information from earlier tokens. In the standard approach, the model compares a current token’s query with other tokens’ keys, turns those comparisons into weights with softmax, and uses the weights to combine the tokens’ values. Positional information is added separately. Softmax weights always add up to one, even when none of the available tokens is a good match.

The authors ask whether choosing relevant content, accounting for position and normalizing the result could arise together from a physical-style process. Their analogy is a radio: a station carries a signal, while a receiver selects what to hear and tunes to a frequency.

A receiver for every token

In the proposed “radio head,” each past token becomes an oscillator: a mathematical object that carries information and changes phase over time. Its content determines how strongly it can influence the current token, like a signal’s compatibility with an antenna. Its age determines its frequency. The current token acts as a receiver, with a dial that can tune to a particular age.

The receiver follows a synchronization model related to the Kuramoto model, used in physics to describe how oscillators can fall into step. The researchers derive a closed-form attention rule from this model. When the receiver locks into a stable state, the rule describes its weights directly. When it is below the locking threshold, the receiver does not settle: it continues to orbit, and the layer reads its time-averaged amplitudes instead. The model calculates these readouts rather than simulating oscillator dynamics each time it processes text.

A key feature is the threshold. When coupling to the context is strong enough, the receiver locks in and its weights normalize, summing to one. Below the threshold, it remains in a “silent” phase. Its weights sum to less than one, allowing the layer to return less information rather than distribute all its attention across poor matches. The paper connects this lower-weight state to abstaining and counting repeated tokens.

The derivation relies on averaging out interactions, or “beats,” between tokens. The authors describe this as the theory’s main approximation. It becomes more accurate for wider attention heads, where token representations are less likely to overlap strongly. Their analysis and simulations also show larger errors in some slow or unresolved heads, where tokens cannot be cleanly distinguished by frequency.

What they tested

The team compared the derived rule with simulations of the oscillator dynamics. They then trained small transformer models using the new attention layer and compared them with models using softmax attention and another synchronization-based design. Tests covered synthetic recall, counting and copying tasks, as well as next-token prediction on TinyStories, a dataset of short stories.

The results suggest the two phases can serve different functions. In recall tests, trained receivers could lock onto a matching key and remain silent when a queried key was absent. In counting tests, repeated copies could add to the total weight while the receiver remained below its threshold; the count was lost when it crossed into the locked phase. In copying tests, performance fell when the sought token’s age moved outside the receiver’s predicted locking range. In these experiments, the authors report that the predicted boundaries tracked where the models failed.

On TinyStories, the radio-head layer approached the next-token accuracy of softmax as head width increased. In the paper’s longer-context tests, its performance held up better than the tested softmax and synchronization-based alternatives. These tests used short-story data and deliberately extended context beyond the training length. They do not show that the method will solve long-context use in deployed language models.

What remains uncertain

The strongest caveat is scale. The language experiments used small, two-layer models trained on TinyStories, with contexts of 128 tokens during training. The authors explicitly leave open whether the same picture holds in large models trained on varied, long-form text. Synthetic tasks help isolate mechanisms, but cannot establish broad language ability or reliability.

The tested implementation also does not reduce computation: its cost is quadratic in context length, like softmax, and the paper reports that its runs took longer. A physical implementation using oscillators is suggested as a possibility, not demonstrated as a practical or faster system. Comparisons are limited to the models, tasks and training setups reported; they should not be read as a general ranking of attention methods.

This article is based on an arXiv preprint dated November 2026. Preprints may not have undergone peer review, and the claims need independent replication, especially at larger scales. The contribution is a theory and a small-model demonstration: synchronization offers a way to combine content, position and the option to stay quiet, but its value for large language models remains an open question.