S-JEPA learns several possible answers to a hidden image region

A hidden patch in a street scene might contain a pedestrian, a traffic signal or something else consistent with its surroundings. Predicting one answer does not describe those alternatives. In a preprint, Nikos Giakoumoglou, Paschalis Giakoumoglou and Tania Stathaki introduce S-JEPA, a system designed to predict a range of possible representations of missing image content.

The reported results suggest that this added variability can coexist with useful image features. However, the benefits depend on the task and model settings, and the uncertainty signal is not a reliable map of where an individual prediction goes wrong.

Predicting features rather than pixels

S-JEPA builds on Joint-Embedding Predictive Architectures, or JEPAs. These systems learn from images without using category labels during pretraining. One network processes visible image patches, while another supplies numerical representations of hidden target patches. A predictor learns to estimate those target representations from the visible context.

These representations, often called embeddings, describe learned image features rather than raw pixel colours. Predicting them lets the training objective operate without reconstructing a photograph.

The deterministic I-JEPA baseline produces one prediction for a given context and target location. S-JEPA adds a sampled hidden code that can change the predicted content while the visible context stays fixed. Earlier related approaches introduced randomness into positions; this work instead targets uncertainty about what a hidden region contains.

How the system learns alternatives

During training, one small network sees both context and target representations and learns a distribution over hidden codes. A second network sees only the context and learns to anticipate those codes without seeing the answer. At inference, when the target is unavailable, samples come from this context-only network.

Training balances two requirements: accurately predicting the target representation and keeping the two code distributions close. That second requirement limits a shortcut in which the training-only network simply passes along target information unavailable at inference.

S-JEPA also retains the slowly updated target network and blocked target gradients used to stabilise learning. Experiments removing these safeguards show that stochastic prediction does not eliminate their importance.

The authors describe the objective as a probabilistic generalisation of the I-JEPA loss. This is a statement about the mathematical form of the loss, not proof that S-JEPA exactly contains the original I-JEPA architecture.

Modest gains, with exceptions

The main experiments pretrained two sizes of vision transformer on ImageNet-1K for 600 passes through the training data. To assess their learned features, the researchers froze the image encoder and trained a simple classifier on top.

S-JEPA reached 73.5% accuracy with the smaller model and 78.3% with the larger one, compared with 72.9% and 77.5% for I-JEPA. It also improved results when the full model was fine-tuned using just 1% of ImageNet labels.

Transfer results were mixed. S-JEPA improved object and scene classification and synthetic object-counting and depth tasks, but trailed I-JEPA on fine-grained species classification. Smaller-scale experiments also found settings where I-JEPA performed better, including shallow predictors and grid masking.

The main benchmark results came from single training runs; supplementary experiments averaged three seeds. Small differences therefore warrant caution. S-JEPA also exceeded a reported LeJEPA result at the same encoder architecture, but used a longer training schedule and more total encoder-token processing. That is not an equal-compute comparison.

What the uncertainty does—and does not—show

Repeated samples produce different predicted representations. For visual examples, the researchers trained a separate diffusion decoder to turn these representations into images, keeping its noise seed fixed across samples. The examples showed changes in object identity and arrangement. This decoder is a visualisation tool, not part of S-JEPA’s core pretraining objective.

The researchers also measured how much predictions varied at each patch. Under masked evaluation, this variability correlated with prediction error across pooled patches, but the average correlation within individual images was much weaker. Attractive uncertainty overlays should therefore not be mistaken for dependable error localisation.

Raw uncertainty estimates were too narrow. Rescaling their spread using a separate calibration split improved agreement with observed errors, but does not establish reliability beyond the tested setting.

A single prediction added little measured inference overhead. Uncertainty estimation, however, requires repeated predictor passes, and image completions require the separate decoder. The method also uses one shared hidden code per image rather than separate codes for each target region.

The preprint’s contribution is thus narrower than general visual understanding: it demonstrates a way to add varied predictions and an imperfect uncertainty signal to representation learning, with modest benchmark gains under several—but not all—tested conditions.