LEF-JEPA targets object structure without preserving every visual detail

A visual model can retain abundant texture and color information without necessarily becoming better at identifying object boundaries or supporting action planning. In an arXiv preprint submitted on November 6, 2026, Zhiyuan Liu, Zhihao Xu, Sicheng Liu and Huanhuan Huang introduce LEF-JEPA, a method intended to preserve useful spatial structure without overemphasizing low-level appearance details.

The approach builds on joint-embedding predictive architectures, or JEPAs. These models learn to predict internal features of hidden image regions rather than reconstructing every missing pixel. The authors argue that supervising prediction only at masked positions can leave object shapes and boundaries poorly defined. They also report that stronger spatial supervision in V-JEPA2.1 retains substantial texture and color variation, motivating a different training signal.

LEF-JEPA keeps masked-feature prediction and adds a recovery task within the model’s latent space: its internal numerical representation of an image. Visible image patches are divided into two groups. One supplies candidate features to be corrected; the other supplies contextual anchors. A slowly updated target encoder processes the unmasked image, combining features from several layers to provide a reference containing intermediate spatial information and higher-level semantics.

The method perturbs candidate features with Gaussian noise, then trains an energy function to guide them toward that reference. Here, energy is a learned score, not a measurement of electricity consumption. Lower scores identify configurations favored by the learned function. Recovery steps are clipped, and their supervised movement is limited to a reachable distance, rather than demanding complete correction when the allowed steps cannot achieve it.

A separate objective transfers this structural preference into the visual encoder itself. It penalizes encoder outputs whose energy exceeds that of the target reference, without forcing the two representations to match exactly. Training the energy function and updating the encoder follow separate gradient paths: encoder updates do not differentiate through the multi-step recovery procedure. After pretraining, the energy module is discarded, so it adds no downstream inference overhead.

The researchers tested two implementations. The first uses a neural energy function, with three local recovery updates in the main experiments. The second, LEF-JEPA-CIM, predicts a conditional Ising model involving 16 logical binary spins and uses a physical photonic coherent Ising machine to search for low-energy configurations. Those discrete configurations are rescored using differentiable coefficients, allowing them to contribute to neural-network training. Each recovery trajectory queries the hardware once and reuses its returned candidates for later local updates.

All methods were pretrained from scratch on unlabeled CIFAR-10 images under matched settings within each encoder size. Training used 32-by-32-pixel inputs and two-pixel patches, producing a 16-by-16 feature grid. The researchers tested small and base vision transformers. After pretraining, encoders were frozen while matched downstream heads or predictors were trained for CIFAR-10 classification, PartImageNet foreground segmentation and goal-reaching in a controlled Two-Room environment. Results aggregate three independently pretrained encoders; planning used 100 episodes per encoder.

Both LEF implementations exceeded the retrained V-JEPA2.1 baseline on all three tasks at both encoder sizes. With the base transformer, LEF-JEPA-CIM achieved 92.27% classification accuracy, 72.11% foreground intersection-over-union and 86% planning success. V-JEPA2.1 scored 88.77%, 67.47% and 77%, respectively. Intersection-over-union measures overlap between predicted and reference foreground regions. With the small transformer, neural LEF-JEPA achieved the strongest foreground segmentation among the main configurations, at 70.04%, compared with 65.30% for V-JEPA2.1.

The hardware version’s reported forward latency was approximately 45–47 milliseconds with one local update, below the neural version using three. However, that measurement excluded network transmission, service queueing and result-return latency. It therefore does not establish a faster end-to-end cloud training workflow or lower total energy consumption.

The findings remain limited to this experimental setting. PartImageNet images and masks were resized to 32 by 32 pixels, and planning involved open-loop navigation through a doorway rather than real-world robotic control. Ablations also showed sensitivity to the strength of the encoder-energy constraint, with overly strong settings damaging classification and segmentation. The preprint supports further investigation of latent recovery as spatial supervision; long-video world models and larger specialized solvers remain future work.