How action-free human videos could help robots learn

Learning from video without action labels

Robot policies use camera images and instructions to choose movements. Training commonly draws on robot demonstrations that pair what the robot sees with what it does. Human egocentric video, by contrast, may lack action labels. Lin Long and colleagues investigate whether it can still contribute to robot policy learning without reconstructing human actions.

Their method, Shared Prediction on Action and Representation (SPAR), starts from a pretrained vision-language model and adds two prediction branches. One predicts robot action sequences; the other predicts features of a future visual frame. Robot data trains both branches, while action-free human video trains only the visual-prediction branch. The branches do not attend to each other. The authors’ design routes the video-based learning signal through the shared model backbone, though that design alone does not establish the sole explanation for any measured performance changes.

The visual target is a learned feature representation, not necessarily a pixel-by-pixel recreation. The model is trained to anticipate a future visual state. At deployment, the visual-prediction branch is removed and the action branch produces movements. Robot action demonstrations remain part of training: human video adds visual supervision, rather than robot action labels.

Results in simulation and on a physical robot

In simulated LIBERO-Plus evaluations involving distribution shifts, the choice of visual target affected performance. In the reported comparison, several learned visual representations outperformed a reconstruction-oriented video latent. SigLIP2 features had the highest reported average success, 76.2%, compared with 60.0% for the Wan-VAE latent. These figures describe this study’s setup, not a general ranking of visual encoders.

The authors also compared training objectives. Direct regression to future features performed broadly comparably to a diffusion-style objective for several tested targets. In one SigLIP2 experiment, predicting one future frame yielded a reported average success of 74.32%, versus 74.62% for four frames. That close result suggests one frame can work in this setting; it does not show that frame count never matters.

Video selection also mattered. The team split Ego4D footage into clips and applied two filters: one favored clips with stable motion estimates, and another required a visible hand. Together, the filters retained about 975 hours from approximately 3,976 hours of raw video. In the reported comparison, raw clips performed worse than the robot-only baseline, while clips passing both filters did better. This suggests that, for this method, the content and quality of added footage matter.

In a separate scaling experiment, the authors increased Open-AoE video from zero to 1,500 hours. In-domain LIBERO success changed little; LIBERO-Plus success rose with video hours. This is evidence of a trend on that out-of-distribution benchmark, not proof of general improvement across tasks or settings. In a fixed-budget comparison, increasing scene, object, or interaction diversity did not produce consistent gains.

The physical-robot test used five bimanual manipulation tasks and 50 trials. The authors’ policy averaged 40% success, compared with 44% for the π0.5 comparison policy. They tied on three tasks—cucumber transfer, toaster loading, and bottle uncapping—and differed by one trial on flower handover and cup hanging. Cucumber transfer and flower handover had relatively strong results compared with the other tasks; both policies rarely completed cup hanging, bottle uncapping, or toaster loading. Precision grasping, bimanual rotational manipulation, and long-horizon execution remained difficult. The paper also notes that the proposed policy used a smaller robot pretraining corpus and trained its action expert from scratch, so the comparison does not isolate the effect of SPAR alone.

Under visual perturbations, outcomes depended on the task. On cucumber transfer, the authors’ policy did better than π0.5 under the tested lighting, background, and distractor changes. On flower handover, it did worse under illumination changes and static distractors, and matched the comparison under background variation. Each condition involved 10 trials per task, limiting what these results can establish beyond the tested setup.

What the study does—and does not—show

The findings suggest that action-free human video can provide useful visual supervision in this training approach. They do not show that such video replaces robot demonstrations: robot data trains the action branch, and robot demonstrations are used for downstream fine-tuning. The authors suggest that the smaller robot pretraining corpus may be related to grasping failures, but this is not established as the cause.

This article is based on the arXiv version 1 preprint, which identifies itself as a preprint; the supplied material does not establish peer review. Results come from particular models, datasets, filtering choices, and tasks, including a limited physical-robot evaluation. Testing across additional robots and settings could clarify how broadly the findings apply; that is a prospective question, not a result reported here.