AuroraWorld turns outside-camera footage into synthetic robot-eye videos

A robot learning to manipulate objects benefits from seeing the action up close. Wrist-mounted cameras capture contact between grippers and objects, along with changes that may be difficult to observe from across a workspace. But collecting these views requires additional equipment and platform-specific data collection. A new preprint introduces AuroraWorld, a system designed to generate those missing perspectives from external-camera videos.

The authors frame the problem as more than changing a camera angle. A useful synthetic video must preserve object positions, follow the robot’s movements and show interactions at the right moment. They argue that combining these constraints explicitly can produce training data that is more useful than visually plausible footage alone.

Building a scene that changes over time

AuroraWorld organizes its internal representation around three ingredients: spatial structure, temporal anchors and motion trajectories. The researchers call this a structured 4D world representation, referring to three-dimensional scenes evolving over time. It does not require an explicit, fully reconstructed 3D scene.

First, an adapted inverse dynamics model estimates the robot’s end-effector trajectory from external observations. Each estimated state contains position, orientation and gripper state. This supplies a motion guide and helps predict the viewpoint of a camera moving with the robot.

Second, the system chooses informative moments when the end effector has changed position or orientation and then settled. A view-synthesis model generates robot-eye images for these moments. These synthetic keyframes act as temporal anchors, constraining what the final video should show at particular times. The detailed implementation uses three interior anchors alongside the first and last frames.

Third, the system extracts learned scene features that encode geometry from the external views. These features and the estimated motion shape the starting representation for a diffusion-based video generator. Rather than beginning with unstructured random noise alone, generation starts with a mixture containing scene and motion information. The synthetic keyframes provide additional guidance.

Testing the footage—and its usefulness

For real-world video generation, the researchers selected 17,000 successful manipulation episodes from DROID and reserved another 200 for evaluation. They also tested robot learning in the 18-task RLBench simulation benchmark and collected 600 demonstrations across six tabletop tasks using an ARX X5 robot.

On DROID, AuroraWorld using inferred trajectories outperformed the six comparison methods on four of five reported video metrics. Those measures assess aspects of pixel accuracy, structural resemblance and perceptual or distributional similarity. It did not lead every measure: Boundless World Model, or BWM, achieved a slightly better Fréchet Video Distance, a measure of differences between generated and real video features.

The comparison methods did not all receive identical inputs. BWM, for example, received the first recorded egocentric frame, while AuroraWorld generated its target-view anchors. These differing conditioning protocols are important context for interpreting the results.

The more practical question was whether synthetic views helped robots learn. With recorded action supervision on RLBench, the RVT-2 policy achieved 71.89% average success using AuroraWorld videos, compared with 70.78% without egocentric training observations and 72.49% with recorded egocentric views. For the π0.5 policy, the corresponding results were 43.56%, 38.96% and 44.78%.

The researchers also tested an eight-task simulation setting in which both the synthetic views and action targets for the converted demonstrations came from external recordings. AuroraWorld produced the highest average success among the compared generators for both policy architectures. This does not mean the entire system learned without recorded robot-eye data: separate demonstrations containing those views were used to adapt the generators.

Real-robot gains, with a precision gap

On the ARX X5, policies trained with AuroraWorld-generated views achieved 44.7% average success across six tasks, versus 40.0% with BWM-generated views and 58.7% with recorded egocentric observations. Each condition received 25 physical trials per task.

Cup stacking exposed a sharp limitation: the AuroraWorld-trained policy succeeded in none of its 25 trials, while the recorded-view condition succeeded in 12. The authors associate this gap with the precise object alignment and contact understanding the task requires.

Crucially, the generated videos were used only for offline training. During evaluation, policies using egocentric inputs received simulator-rendered views or live wrist-camera images. The study therefore supports filling gaps in training datasets, not removing wrist cameras from deployed robots.

The authors also acknowledge that errors in inferred motion can propagate into camera predictions, anchor selection and generation. Severe occlusion and rapid contact changes can leave external footage insufficient to determine the missing view. AuroraWorld offers evidence that structured synthetic perspectives can improve robot learning, but its results also show that convincing generated observations are not interchangeable with accurate recorded ones.