H2RB preprint explores how human demonstrations can teach robots new tasks

Two gaps between watching and doing

Teaching a robot through demonstrations requires more than showing it the right sequence of movements. Human and robot observations look different, and their hands and arms realize actions differently. The authors argue that putting both kinds of motion into a common coordinate system leaves important differences unresolved.

Their paper, Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs, was posted as an arXiv preprint on November 3, 2026. It presents experimental findings, not an established demonstration of general-purpose learning from human video.

H2RB adapts a pretrained vision-language-action model: a policy that uses images, instructions and the robot’s physical state to predict actions. The framework combines human and robot demonstrations without requiring matching recordings of each performing the same trajectory.

Shared structure, separate movement mappings

The method first reconstructs wrist motion and hand articulation from human videos, converts them into the robot’s coordinate convention and retargets the hand movements. Human and robot actions then share a representation containing wrist position, wrist orientation and hand configuration.

A shared encoder encodes action sequences into shared latent representations—internal numerical descriptions used as prediction targets. Separate human and robot decoders reconstruct movements from that space. The intention is to share useful behavioral structure without requiring both sources to follow identical motion dynamics.

A second component connects the model’s visual-language features to physical hand geometry. During training, an adapter is supervised using the current wrist pose and five fingertip positions. Each observation supplies its own geometry target, so this process does not require paired human–robot examples.

For the main transfer experiments, deployment uses the human-trained decoder. Its output is already in a robot-compatible wrist–hand representation; inverse kinematics converts wrist poses into arm commands. This is therefore not direct imitation of untreated video.

What the robot experiments measured

The controlled experiments used an OpenArm platform with dexterous hands, operating its right arm and hand. All policies started from the same robot-pretrained RLDX-1-IMG checkpoint.

In object transfer, human demonstrations introduced box, ball and towel pick-and-place tasks, while robot demonstrations supplied experience with other objects. In skill transfer, humans demonstrated opening and closing an air fryer, while the source robot demonstrations contained pick-and-place tasks instead.

The authors report an overall object-transfer score of 51.8 for H2RB, compared with 8.0 for basic human–robot co-training and 27.2 for an Ego-Pi-based shared-action baseline. Under object-placement shifts, the corresponding scores were 45.3, 6.0 and 21.3.

For opening and closing, H2RB scored 62.7. Basic co-training and both shared-action baselines scored zero on those target skills. Human-only adaptation nevertheless scored 29.0, so the zero-transfer finding does not apply to every comparison method.

Crucially, these are weighted task-progress scores on a 0–100 scale, not completion percentages. Reaching, grasping and partial execution earn credit. The appendix reports full completion in 12.2% of H2RB’s object-transfer trials and 30.0% of its skill-transfer trials. Evaluation comprised 90 target-object rollouts and 30 target-skill rollouts per policy.

Additional training and a second robot

Additional heterogeneous training across human and robot datasets raised H2RB’s object-transfer score to 57.2 and skill-transfer score to 81.7. Full completion reached 21.1% and 56.7%, respectively.

A separate modification repeatedly reintroduced instruction features inside the action model, increasing the skill-transfer score from 62.7 to 83.3. That experiment was distinct from the heterogeneous-training comparison; the paper does not present these gains as a combined result.

On an RB-Y1 robot, the authors also tested parcel handling after robot-only downstream fine-tuning. Binary success was 64% following robot-only heterogeneous training, 72% following human–robot training and 78% when that human–robot training included geometry grounding. Each policy received 50 trials. Unlike the OpenArm scores, these percentages required complete task success. This experiment did not use the shared action tokenizer during downstream fine-tuning.

What remains uncertain

The controlled demonstrations were collected in the same environment, with approximately matching human and robot camera viewpoints. Placement shifts test a specific form of generalization, not unrestricted changes in scenes, hardware or tasks. Each reported condition used one independently trained policy, leaving variation across repeated training runs unmeasured.

H2RB also depends on motion reconstruction, retargeting and existing robot experience. Avoiding robot demonstrations of the target behaviors is not the same as eliminating robot data. Its tokenizer requires a separate training stage beyond policy fine-tuning.

The results support the authors’ narrower proposition: explicitly aligning perception and action representations can make human demonstrations more useful for adapting a robot-pretrained policy. They do not establish reliable task completion across arbitrary human videos or deployment environments.