NVIDIA adapts Cosmos models for interactive robot and driving simulation

NVIDIA’s preprint, “Interactive World Simulation with Cosmos for Physical AI,” submitted to arXiv on November 4, 2026, describes a shared framework for adapting Cosmos 3 world models to ongoing interaction. Rather than generating only a complete video, the specialized models produce observations incrementally as actions or scene controls become available.

The report distinguishes two roles. A model can predict the visual consequences of supplied actions, or it can generate observations from physical state maintained by an external simulator. In the latter case, the simulator remains responsible for evolving the physical state; the neural model supplies its visual appearance. The authors demonstrate three specializations, with different controls, benchmarks and runtime requirements.

Teaching a video model to continue

The shared training recipe has four stages. First, a model learns its application domain with access to complete training clips. Next, causal training teaches it to predict successive chunks using earlier recorded observations, without reading future target chunks.

Consistency distillation then reduces generation to four denoising evaluations per chunk. Finally, self-forcing distribution matching trains the student on rollouts containing its own earlier predictions. This addresses the mismatch between learning from clean recorded history and continuing from imperfect generated history.

During streaming execution, cached attention features preserve selected initial and recent observations. Bounding this history limits memory and attention costs, but also limits direct access to older observations. Compilation and other runtime optimizations reduce repeated work; updating the history and decoding observations still contribute substantial costs.

One model for navigation and manipulation

Cosmos3-Nano-Sim-Bimanual predicts RGB observations from an initial image, language and actions. Its shared action interface covers camera movement and robot manipulation, allowing switches between those control modes without restarting the visual history.

The authors report stronger video-quality scores than the listed navigation baselines across three benchmark groups. Camera following is not uniformly better: on PAI-Bench-G, absolute translation error is higher than LingBot-World 2.0’s, despite improvements in relative motion metrics. On two AgiBot robot benchmarks, the model improves all five reported quality and reference-image agreement measures relative to the listed DreamDojo teachers.

These tests replay recorded or synthesized actions. They do not test a policy selecting its next action from generated observations, and the image-agreement metrics do not independently validate contact mechanics or task success.

On one GB300 GPU, a compiled configuration produces 640 × 352 video at a reported steady capacity of 48.1 frames per second. Its eight-frame chunks take a median 166 milliseconds. That capacity does not include every source of interactive response delay, such as collecting actions for a chunk or displaying the output.

Rendering inside a robot feedback loop

Cosmos3-Nano-Sim-Transfer generates RGB from language and one spatial-control stream: depth, segmentation, edges or blur. An external simulator can supply those controls while maintaining the physical scene.

On TransferBench, the final student improves all four spatial-control metrics relative to the Cosmos 3 GA reference, but results are mixed elsewhere. Technical video quality and prompt agreement decline slightly, while quality drift and temporal geometric error increase. The adapted bidirectional teacher remains stronger across all nine reported metrics.

The report also demonstrates an actual policy feedback loop in Isaac Lab. A Unitree G1 apple pick-and-place policy receives either native Isaac RTX images or depth-conditioned generated images. At the 15,000-step policy checkpoint, completion rises from 22 of 32 matched starts to 27 of 32—68.8% to 84.4%.

However, the appearance prompt was calibrated using those same starts and checkpoint. Other appearances and policy checkpoints produce mixed outcomes, including declines. The policy was fine-tuned only on real demonstrations; this experiment evaluates generated observations rather than demonstrating policy learning from them.

Driving sensors and the limits of speed

Cosmos3-Nano-Sim-Auto extends generation to synchronized cameras and LiDAR. Camera-aligned World Scenario Map controls specify road structure and actors, while HD-map controls guide LiDAR. After startup, each chunk spans 0.4 seconds: 12 RGB frames per camera and four LiDAR sweeps.

Selected configurations reach real-time throughput, but this is not universal. One camera on one GB200 reaches a median presented rate of 32.6 frames per second. Seven-camera RGB generation reaches 23.2 on 32 GPUs; seven cameras plus LiDAR reach 16.7 on 16 GPUs, below the 30-frame-per-second video rate. Some configurations also prefetch controls with a one-chunk delay.

The causal driving model improves several distributional and control-following measures over the listed baselines, while sacrificing technical quality and some sensor fidelity. Hour-long tests cycle existing controls and captions; stable aggregate scores coexist with possible deterioration in object appearance and physical plausibility.

The contribution is therefore a shared method for controllable, continuing observation generation—not blanket validation of simulation accuracy or deployment readiness. The report lists open-source code and open-weight checkpoints for further development and evaluation.