What should an AI explore before it has a task?

What should an artificial agent do before anyone tells it what to achieve? Random movement collects experience, but not necessarily experience that helps with later goals. A study asks whether internally generated rewards can guide more useful exploration.

In the November 9, 2026 arXiv preprint Building World Models from Intrinsic Motivation, Reece Keller and colleagues compare 24 exploration objectives within a shared learning framework. Their strongest overall pattern favors curiosity combined with reconstructing observations. But the best individual combination changes across experiments. These are preliminary results from simulated environments, not a demonstration of general intelligence or real-world robotic competence.

Measuring exploration by what it teaches

A world model predicts how an environment changes when an agent acts. Instead of learning every behavior solely through direct interaction, an agent can train a policy—its rule for choosing actions—using trajectories generated inside that model.

The researchers build on DreamerV3 and organize exploration incentives into three families. Curiosity rewards surprise, uncertainty or improvements in prediction. Entropy objectives encourage visits to less familiar states. Competency objectives favor actions or skills that produce reliably distinguishable outcomes.

These are computational rewards, not evidence that the agents experience human curiosity or motivation.

The study also compares three ways to learn the model’s internal representation. Standard DreamerV3 learns to reconstruct observations, preserving information needed to reproduce what it sees or senses. NE-Dreamer predicts the next observation’s compressed representation. R2-Dreamer aligns the current model state with the current observation’s representation while discouraging redundant features. Their auxiliary architectures and loss scales differ, so this is not a perfectly isolated comparison of one mathematical objective.

During pretraining, intrinsic rewards guide exploration without directly changing the world model’s representation objective. They influence the model through the experiences collected. Task rewards can separately train an evaluation reward predictor and task policy, but cannot shape the exploration policy or world-model representation.

That distinction matters for “zero-shot” performance here: it means control before collecting task-directed training interactions, not control without access to task rewards or task-specific policy training.

Different environments favor different recipes

The experiments include 25 DeepMind Control tasks using body-state observations, such as joint information, plus visual Walker tasks, simulated kitchen manipulation and Craftax Classic. Craftax requires gathering resources and combining actions into longer achievement sequences. Pretraining uses one million environment steps for the control suite and five million for Craftax.

Curiosity has the strongest overall family performance, especially alongside observation reconstruction. Some intrinsically pretrained agents approach or match task-supervised performance on body-state control tasks. Craftax is harder: the best reported zero-shot result reaches 51% of the task-supervised reference.

Reconstruction-free models nevertheless support strong Walker transfer from body-state observations. Some competency methods also pair better with those models in Craftax. The result is a case for choosing exploration incentives and representations together, rather than declaring one architecture universally superior.

Simple incentives are often competitive. Prediction error measured directly in the world model’s features can match or outperform approaches with additional feature learners. However, the implementations also differ in estimators and settings, so those comparisons do not isolate complexity itself as the cause.

Memory helps, but its cost matters

For adaptation, the researchers retain the pretrained model and stored experience but reset the task policy, value estimator and reward predictor. This tests transfer without simply inheriting an already trained task policy, although benefits can come from both the model and the retained experience.

In Craftax, Never Give Up, or NGU, paired with DreamerV3 performs best on both reported adaptation measures. NGU combines novelty relative to recent states with a longer-term novelty signal. In this implementation, its short-term memory is restricted to the supplied trajectory context.

After 500,000 task-directed steps, that pairing scores 15.8, compared with 5.9 from scratch. The authors estimate that scratch training needs about 3.11 million task-directed steps to reach the adapted score: a 6.22-fold task-interaction advantage.

That is not a sixfold reduction in total training cost. It excludes five million pretraining interactions, offline policy updates and computation. Differences between evaluation protocols also make the multiplier a descriptive estimate rather than an exact efficiency comparison.

More imagined practice is not always better

In visual Walker tasks, additional policy training inside a frozen model improves both initial control and later adaptation. In Craftax, more imagined training can worsen adaptation, even when initial performance improves.

The authors suggest that policies may become committed to familiar behavior supported by an incomplete model. That explanation remains a hypothesis, not an isolated causal finding.

The practical lesson is narrower than “curiosity solves learning”: exploration should be judged by the reusable knowledge it produces. These experiments show promising combinations, while leaving real-world transfer, broader sensory inputs and better long-term memory open.