Can AI read cosmic structure without becoming overconfident?

How much matter does the universe contain, and how strongly does that matter cluster? Weak gravitational lensing helps cosmologists investigate these questions by measuring the subtle, coordinated distortions of distant galaxy shapes as their light passes through intervening matter. But extracting an answer is only half the task. A reliable method must also estimate how uncertain that answer is.

The FAIR Universe Weak Lensing Machine Learning Uncertainty Challenge puts those requirements together. In a November 2026 preprint, Biwei Dai, Po-Wen Chang and collaborators describe the benchmark and results from its first phase, a NeurIPS 2025 competition. The study evaluates inference methods on simulations; it does not report a new measurement of the actual universe.

A controlled sky, with complications

The researchers built simulated convergence maps: two-dimensional representations of the matter distribution projected along the line of sight, weighted by its lensing effect. Their simulations follow the geometry and source-galaxy distance distribution of part of the Hyper Suprime-Cam Year-3 survey.

The training collection contains 25,856 maps, each measuring 1,424 by 176 pixels. Those images cover 101 cosmological models, with 256 realizations per model. Two quantities distinguish the cosmologies: Ωm, the matter-density parameter, and S8, a combination describing the amplitude of matter fluctuations.

The maps also vary effects that can complicate cosmological interpretation. These include the redistribution of matter by processes associated with ordinary matter, such as feedback from active galactic nuclei, and uncertainty in the inferred distance distribution of source galaxies. Participants needed to account for those variations rather than mistake them for changes in cosmology.

The released training maps contain no shape noise, allowing teams to add fresh noise during training. The prescribed noise represents random galaxy-shape orientations at a source density chosen to resemble the sensitivity expected from Euclid and Rubin, rather than that of the current HSC survey. The benchmark therefore combines HSC-based geometry with a future-survey-like noise assumption.

Why the winner depended on the test

Teams had to predict both cosmological parameters and a one-standard-deviation uncertainty for each. The scoring rule balanced prediction errors against the reported uncertainty, with an additional penalty for inaccurate central estimates. Simply widening every uncertainty interval was not a route to success.

The public test contained new map realizations from the same 101 cosmologies used for training. The final evaluation separated two questions. Holdout A contained 6,000 new realizations from those familiar cosmologies. Holdout B contained 4,020 realizations from ten cosmologies absent from training.

That distinction changed the ranking. Cmbagent led on familiar cosmologies, scoring 11.703 on A. Shubhojit led on unseen cosmologies, scoring 11.361 on B, and also topped the aggregate ranking, which averaged the two holdout scores. These are competition scores, not percentages of accuracy.

The authors caution against overinterpreting small differences on B because it contained only ten unseen cosmologies. They nevertheless report a clear lead for Shubhojit there, with the remaining listed submissions clustered more closely together.

Reported uncertainty coverage was generally near the expected one-standard-deviation level. That is encouraging within this experiment, but it is not proof that the same uncertainty estimates would remain reliable on real observations.

Several routes beyond a basic network

The strongest submissions used different combinations of learning methods and cosmological knowledge. Shubhojit used compact image-processing neural networks in ensembles, with an adjustable training objective to balance parameter accuracy and uncertainty estimation.

The Transatlantic Dream Team generated inexpensive approximate simulations, then trained a transformation to make them more closely resemble the challenge simulations. Pretraining on those additional maps, followed by training on the challenge data, contributed to second place overall.

THUML combined a vision transformer with established cosmological descriptors, including the power spectrum and counts of peaks in the maps. Cmbagent used language-model-assisted development to propose and implement methods, alongside further model and human refinements. Its success was not evidence of an entirely autonomous discovery process.

Generalization still has boundaries

Even the unseen-cosmology holdout used the same simulation pipeline and parameter priors as training. It tested generalization within that framework, not resilience to every mismatch between simulated and real skies.

The benchmark also covers just one sky region and one source-distance bin, supplies reconstructed maps rather than full galaxy catalogs, and omits some important observational and astrophysical effects, including intrinsic alignments and point-spread-function effects.

The authors report an expanded training release and a larger test set spanning 82 unseen cosmologies, with that test set reserved until after the second competition phase. Larger areas and more realistic systematics remain planned extensions. For now, the clearest lesson is methodological: a strong leaderboard result needs a test that changes the underlying cosmology, not merely the image noise.