HALO humanoid learns household movements in simulation, then tackles unfamiliar rooms

What the researchers report

In the November 4, 2026 preprint Perceptive Humanoid Motion from Simulation, Siyuan Hu and colleagues at Apple describe HALO: Humanoid Autonomy, Learned in simulation, running Onboard. The system uses camera observations and language instructions to guide a Unitree G1 humanoid through six kinds of physical task.

The authors report 91% average success across 360 single-task trials in six real environments that were not used for training or scanned beforehand. That result covers walking to named objects, sitting, passing sideways through partly open doors, crouching or crawling under tables, and pushing boxes. It is not a measure of unrestricted household autonomy. The findings are reported in a preprint, rather than established through peer review.

Building practice episodes without acting out every run

HALO addresses a training-data problem: robots need examples connecting what they see and what they are asked to do with detailed body movements. Producing every episode through human teleoperation ties data collection to an operator’s time.

The researchers instead assembled approximately 300 hours of paired images, instructions and motion. The appendix specifies 162,698 episodes totaling 296.3 hours. Training scenes were procedurally generated, with varied furniture, layouts and appearances, rather than reconstructed from the eventual deployment rooms.

A motion-generation model produces trajectories from text, navigation paths and key body poses. These movements are placed within simulated scenes and replayed to render a robot’s-eye view. Lighting and image changes broaden the visual conditions represented during training. Geometric checks score or filter motion candidates, although the appendix makes clear that filtering differs between skills and does not eliminate every imperfect trajectory.

This is not a pipeline without human contributions. People design each skill’s motion sequence and constraints. Its ingredients also include a model trained on human motion, recorded pushing motion, keyframes derived from phone videos, and a seated pose from a real robot log. The automated element is producing many situated training episodes, not inventing a new repertoire independently.

Three layers connect instructions to movement

HALO divides control into three timescales. At the top, a vision-language model interprets a request and selects an ordered sequence of available subtasks. It checks progress and can advance, retry or change the plan.

A motion generator then combines the current subtask with a camera image to predict five seconds of whole-body movement. It refreshes that plan about three times per second, rather than following an entire prediction without looking again. New predictions retain a short prefix from the existing plan, helping avoid abrupt changes while the robot keeps moving.

Finally, a tracker runs 50 times per second. It combines the planned movement with depth images, joint measurements and inertial sensing to issue joint-position targets. This layer handles the physical execution of a plan against nearby geometry.

The system’s default configuration runs these components onboard, using a rear-mounted Mac mini and the robot’s Jetson computer. Cloud models are an optional alternative for high-level orchestration, not a requirement of the local configuration.

Single skills transferred better than longer chains

Each of the six single tasks was tested ten times in each real environment. Reported success ranged from 85% for sitting to 100% for crouching, with the six-task average rounded to 91%. Success required completing the task without falling or getting stuck; the appendix also defines task-specific completion conditions.

The longer sequences had a smaller evaluation: three combinations, each tested ten times in one real environment. Passing through a door and approaching an object succeeded in 100% of those trials. Crouching then sitting achieved 60%, while crouching, approaching a box and pushing it achieved 50%. Their average was 70%, distinct from the headline single-task result.

Failures exposed a transition problem. A successful crouch could leave the robot facing away from the next target. In three of the five failed real-world crouch–approach–push trials, the box never returned to view. Some crouch–sit attempts targeted a sofa instead of the intended chair.

What the evidence does—and does not—show

Controlled simulation experiments on sitting support several design choices. Increasing training environments from one to 1,000 raised success from 43% to 84%. Removing depth input from the tracker reduced success from 84% to 74%. These are simulated sitting results, not separate demonstrations of improved real-world performance across every skill.

The authors identify further limits: new skills need human-authored prompts and keyframes; vision does not directly reveal contact when objects are hidden; and the tracker follows movement plans without knowing their intended effect on the world. HALO demonstrates transfer of a defined skill set to unfamiliar spaces, while its longer-chain failures show that reliable task transitions remain unresolved.