JEPAhead predicts future driving features, but practical benefits remain unproven

From filling gaps to forecasting

JEPAhead asks whether a video model trained to fill missing information can be adapted to predict strictly future observations. Daniel G. P. Petrini and colleagues at Brazil’s Eldorado Research Institute and the University of Campinas describe the work in an arXiv preprint submitted on November 5, 2026. The supplied record identifies version one and does not establish peer-reviewed publication.

The starting point is V-JEPA1, which learns to predict representations of masked video regions. These representations, or latent features, are numerical descriptions rather than images. Filling a gap can use information from either side of that gap; forecasting must respect time’s direction.

The researchers keep the pretrained video encoder—the component that converts images into features—fixed. They fine-tune its predictor to estimate future features from a continuous block of past observations. Their central result concerns prediction in this feature space, not autonomous driving performance.

How the system works

Each example contains 32 observed frames followed by 16 future frames, sampled at four frames per second. The encoder represents pairs of frames as temporal groups, each containing a spatial grid of 196 feature tokens. The predictor receives 16 past groups and forecasts eight future groups in one forward pass.

Future queries specify positions in time and space but contain no future image content. A group-level attention mask constrains information flow in temporal order. The output predicts changes relative to the last observed group, rather than estimating every future feature from scratch.

There is another safeguard against looking ahead: each target representation is calculated from a video window ending at that target group. No frames beyond the target enter its encoding. Training combines losses on feature similarity, feature values and temporal changes.

The model is action-free: it does not receive proposed future steering or acceleration commands. Here, “causal” describes temporal information restrictions, not an ability to establish cause and effect or compare alternative driving decisions.

Testing against an unchanged future

The experiments use driving sequences from CARLA Town12, generated by a rule-based driving expert. Routes were divided into training, validation and test sets so that frames from one route could not appear across different splits. The resulting sets contained 6,368, 931 and 1,773 examples, respectively.

For the main retrieval test, researchers selected 591 temporally non-overlapping test examples. Each prediction was compared with all 591 corresponding futures, asking whether the correct one ranked highest by feature similarity.

JEPAhead succeeded in 69.0% of cases, compared with 55.3% for persistence: repeating the last observed features throughout the forecast. That is a 13.7-percentage-point improvement. A smaller predictor trained from scratch reached 58.8%.

The advantage over persistence increased at longer horizons. At the eighth future group, retrieval accuracy was 51.3%, versus 35.9%. Feature-similarity improvements were also larger among tokens showing the greatest change, an important distinction because much of a driving scene remains relatively static.

The adapted predictor has 23.9 million parameters, against 3.97 million for the compact model. Because these models differ in size, architecture and initialization, their comparison does not isolate the contribution of pretraining. Persistence also achieved the best reported distributional agreement with real features, showing that the learned model did not lead on every measure.

Meaningful features, limited image gains

To examine scene meaning, the team used a fixed linear classifier trained on real encoder features. On patches whose semantic class changed, predicted features achieved 0.16 mean intersection-over-union, a measure of class overlap, compared with 0.13 for persistence and 0.36 for actual future features.

Vehicle-conditioned retrieval improved over persistence by 15 percentage points at the longest horizon. This supports the presence of driving-relevant information, but does not establish accurate forecasts of individual vehicles’ trajectories.

A separately trained decoder converted features back into images for evaluation. JEPAhead reduced average absolute pixel error by 3.8% overall and 13.5% on the most-changing pixels. However, LPIPS, a learned perceptual similarity metric, became slightly worse overall and improved only marginally on changing pixels.

Even decoding actual future features produced reconstruction errors. The study therefore cannot resolve how much perceptual quality is limited by forecasting and how much by decoding.

What remains untested

The evidence is confined to one simulated town and a 16-frame forecast. Generalization to real roads and longer horizons remains untested. Without future action inputs, the model cannot directly evaluate competing action sequences for planning. Object-level forecasting, downstream perception benefits and improved driving decisions also remain unverified.

The result is a promising representation-learning experiment—not evidence of a deployable driving system.