MIRA preprint explores a signal-processing approach to recurrent language models

Efficient language models must balance what they remember with the cost of processing longer sequences. A new, non-peer-reviewed preprint proposes MIRA, an architecture inspired by digital signal processing, and tests whether expanding the patterns its recurrent memory can represent improves language modeling.

The proposed limitation: how memory changes over time

In their preprint posted on November 6, 2026, Róisín Luo, Ihsan Ullah, Christian Gagné, Dong Li and Karyn Morrissey examine recurrent alternatives to Transformer attention. These models carry a memory state forward through a sequence rather than relying on the same attention mechanism as a conventional Transformer.

The authors analyze recurrent sequence mixers as infinite-impulse-response, or IIR, systems: systems in which an input can continue influencing later outputs through feedback. In signal processing, mathematical quantities called poles help describe whether that influence fades, persists or oscillates.

Under the paper’s analysis, several existing recurrent designs constrain their poles to the real axis. Complex-conjugate pole pairs offer additional oscillatory modes. Here, “bias” means a restriction imposed by an architecture’s mathematics, not demographic or social bias.

An important qualification appears in the appendix: for input-dependent models, the pole analysis holds coefficients fixed at a token position. The actual sequence response depends on how those coefficients change over time. The paper also notes that RetNet’s evolving positional rotations can produce complex lag factors, beyond its frozen-coefficient characterization. These distinctions limit broad claims that existing models cannot represent particular temporal behavior.

How MIRA works

MIRA stands for Modal IIR Recurrent Architecture. Each block combines a lattice–ladder sequence mixer with a feed-forward network.

The lattice couples the current input with stored states through successive stages. A ladder readout combines information from those stages into an output. Input-dependent controls adjust the dynamics. The structure can represent both real and complex-conjugate poles, while recurrent processing supplies temporal order without explicit positional encoding.

The authors also develop a specialized initializer to distribute initial memory-decay timescales. Their mathematical appendix provides sufficient conditions for stable frozen dynamics and, under a uniform contraction bound, exponentially decaying time-varying dynamics.

For GPU execution, MIRA groups tokens into chunks. It summarizes each chunk, combines summaries to recover boundary states, then replays chunks concurrently to produce outputs. This exploits associative composition while preserving token order. Total work scales linearly with sequence length for a fixed configuration; parallel execution does not make the total computation sublinear.

What the experiments tested

The main model, Mira-S, has 12 layers and 144.72 million parameters. The researchers trained it and similarly sized Transformer, RWKV-6, RWKV-7, Mamba-1 and Mamba-2 baselines on FineWeb-Edu, using a training window of 512 tokens and a weight-update budget corresponding to nine billion tokens.

They evaluated text prediction on held-out FineWeb-Edu and the test splits of WikiText-103 and HellaSwag. Measurements included next-token accuracy and perplexity, where lower perplexity indicates better prediction of the evaluated text—not necessarily better reasoning or more useful generated answers.

A separate synthetic experiment used four-dimensional, 200-step signals generated by a system with nonreal poles. Parameter-matched single-layer models trained for 1,000 optimization steps; the authors report that MIRA recovered the target patterns more accurately. This is a controlled demonstration, not a direct measure of language understanding.

Results: encouraging, but not uniformly best

At the 512-token training length, Mira-S achieved FineWeb-Edu perplexity of 27.75, compared with 29.96 for the Transformer and 30.65 or higher for the recurrent baselines only at the longer evaluation length; at 512 tokens, the recurrent baselines ranged from 31.97 to 35.93. On WikiText-103 at 512 tokens, Mira-S scored 31.03, also the lowest reported value.

The advantage was not universal. On HellaSwag text at 512 tokens, Mira-S’s perplexity was 32.50, behind the Transformer’s 31.00 and Mamba-2’s 31.07. These are text-prediction measurements, not HellaSwag multiple-choice accuracy.

When evaluated at 32,768 tokens—64 times the training window—Mira-S reached perplexities of 25.30, 27.54 and 28.62 on the three corpora, respectively, the lowest values in the main table. Better perplexity at longer contexts does not by itself establish reliable retrieval of every earlier detail.

Smaller, approximately 65-million-parameter ablations also favored lattice orders allowing complex poles over the first-order, real-pole configuration. Larger memories generally helped, but increased computational and memory costs.

Speed claims and remaining limits

On an NVIDIA RTX A6000, the parameter-matched module benchmark reports an 11.37-millisecond forward pass at 32,768 tokens. Reported throughput reached 8.36 million tokens per second: 6.41 times the attention baseline, but just 1.02 times RWKV-6. These are module-level measurements, not demonstrated end-to-end chatbot generation speeds.

The evidence concerns small models, a specific training setup and one reported GPU platform. The supplied text does not report repeated-run uncertainty estimates for the main comparisons. Additional reasoning and retrieval evaluations appear as figures without numerical scores in the extracted text. Overall, the preprint supports further investigation of richer recurrent dynamics, not a demonstrated general replacement for Transformers.