TRANSMISSION // 2611.22257v1
What online LSTMs remember—and where their forecasts can break down
Source published
Simulations in a new preprint show that online LSTM forecasts can depend strongly on model design and learning choices. The authors highlight possible inconsistencies in multi-step predictions and limits to what hidden states reveal about memory.

Long short-term memory networks are built to learn patterns across time. But do they reliably capture long-range memory and produce forecasts that fit together across different time horizons? A simulation-based study of online LSTMs finds that the answer depends on the data, the model and how it is trained.
Long short-term memory networks, or LSTMs, are designed to process sequences. Their internal states are meant to carry information forward, helping the network use past observations when making predictions. In economics and finance, that makes them appealing for data such as market prices and measures of volatility, where what happened earlier may matter now.
But an LSTM forecast is not determined by its architecture alone. The choice of what to predict, how to measure prediction error and how to update the model as new observations arrive can all matter. In this paper, Christian Gourieroux, Joann Jasiak and Cheng Zhong ask how those choices affect online LSTM predictions for stationary time series—series whose statistical properties are assumed not to shift over time.
Forecasts made as data arrive
The authors focus on a setup in which a model processes observations in sequence and updates its parameters using stochastic gradient descent (SGD). In everyday terms, SGD nudges the model’s adjustable settings in response to prediction errors. An LSTM also updates internal quantities called hidden and cell states, which summarize information carried through the sequence.
The study examines how these ingredients work together: the set of forecasts an architecture can represent, the objective function used to train it, and the online learning procedure. It also looks at multi-step forecasts—predictions for several time horizons—and asks whether the internal states behave like useful records of past information.
Rather than reporting a broad evaluation on real-world data, the paper illustrates these issues with simulations. The authors generate time series from several familiar models, including white noise, moving-average processes and a GARCH process, then examine forecasts and model states as the online algorithm runs. They use measures such as forecast errors and plots of the hidden states and their autocorrelation patterns to diagnose behavior.
Different horizons can tell different stories
One central point is that there is more than one way to produce forecasts at multiple horizons. A direct approach trains separate forecasts for each horizon. A recursive approach repeatedly feeds a one-step forecast back into the model to produce longer-range forecasts. These methods need not give the same answers: they use different prediction targets and, in the direct setup, can involve separately updated models.
The authors explain that direct forecasts made independently at each horizon can violate the Law of Iterated Expectations, a rule that links conditional expectations across horizons. Put simply, forecasts made separately for tomorrow and the day after may not agree with what follows from chaining together the relevant shorter-step forecasts. This is a property of the forecast construction, not evidence that every LSTM in every application will fail this way.
The simulations also illustrate limitations in prediction accuracy and in interpreting the model’s memory. In one moving-average example, the online LSTM’s reported root mean squared error was higher than the paper’s analytically calculated optimal benchmark. In another example, the LSTM’s hidden states displayed memory-like patterns even when the input was white noise. Such patterns caution against treating apparent persistence in an internal state as proof that the model has recovered meaningful long-term structure in the data.
The paper further reports that LSTM-based estimates of conditional volatility can sometimes be negative, although volatility should not be. In the examined GARCH simulation, using a joint setup for predicting the series and its square reduced this problem compared with one moving-average example. That is a result about these particular demonstrations, not a general guarantee that joint prediction solves the issue.
Results depend on the setup
The authors’ broader conclusion is that there is no single LSTM prediction setup that is automatically suitable for every data-generating process and forecasting goal. The model’s representational limits, training objective, learning procedure and input data all affect what it learns. Multi-task training—optimizing predictions for several targets together—does not necessarily resolve the problem.
That conclusion should be read in light of the study’s scope. Its evidence comes mainly from simulations using deliberately simple architectures and selected stationary processes, not a systematic contest across forecasting models or a demonstration on a range of real economic and financial datasets. The authors also note that assessing how far a forecast falls short of the true optimal prediction can be difficult in practice. Their diagnostic approach—tracking errors, hidden states and parameter updates through training—offers a way to investigate behavior, but does not by itself establish that a model will perform poorly in a particular application.
The paper is an arXiv preprint, so its claims should be treated as work in progress rather than as settled evidence. The results do not show that LSTMs are useless. They offer a more specific warning: a model’s ability to process sequences does not ensure that its forecasts are optimal, consistent across horizons or easy to interpret. For anyone using an online LSTM, those properties need to be checked for the chosen data and forecasting task.
- LSTM
- time-series forecasting
- online learning
- stochastic gradient descent
- multi-step prediction
- hidden states
- econometrics
Reviewed cross-references
These connections were reviewed by a model, not independently fact-checked by a human. Similarity does not imply agreement; consult the original papers and the evidence below.
Forecasting Construction Material Prices, One Category at a Time
Shared method: Both papers study LSTM-based time-series prediction: Gourieroux et al. analyze online LSTM prediction properties for stationary inputs, while Lyu et al. evaluate LSTM forecasts of construction-material prices. The shared method is bounded to using LSTMs for prediction; the settings and evidence differ, so the first paper does not validate the second’s empirical results.
Model review: · openai-codex/gpt-6-luna
Evidence and provenance
Selected excerpts, not complete papers. PDF extraction can omit figures or damage equations, tables and reading order; sources may be preprints.
2611.22257v1 — Original paper
Excerpt reference:
static/papers/2611.22257v1.md:48-158
Basis: Paper text
Source SHA-256:9f9e80e40413df3e584485661f842511ceec7e02b0f4be8a257765e2156513982512.09360v2 — Original paper
Excerpt reference:
static/papers/2512.09360v2.md:13-42
Basis: Paper text
Source SHA-256:450c12a2d1f0c6e9e02ab48cdce6c5b4b145e7dbe4a90e26bebeb2350010f6fc
Related reading
Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.
Forecasting Construction Material Prices, One Category at a Time
Source published
Shared keywords: time series forecasting
Forecasting electricity prices as a connected day
Source published
Shared keywords: online learning
Turning news into numerical clues for forecasting
Source published
Shared keywords: time series forecasting
Why Patch Size Changes What Attention-Based Forecasters Can Learn
Source published
Shared keywords: time series forecasting
ElecLoad Puts Events at the Center of Electricity Demand Forecasting
Source published
Shared keywords: time series forecasting