Robots may learn more from human corrections when they do not copy them exactly

What makes someone take over?

A person supervising a robot does not necessarily intervene at the first imperfect movement. They may wait to see whether it recovers, then step in when progress stalls. That difference matters for researchers developing algorithms with simulated human supervisors: an unrealistic model of intervention could favor methods that work less well with actual people.

In an arXiv preprint posted on November 5, 2026, Pranav Teegavarapu, Sriyash Poddar and colleagues investigate both when people intervene and how robots should learn from their corrections. Their proposed method, Q-OIL, uses those corrections to encourage exploration while limiting the influence of imperfect advice.

The team first studied eight participants familiar with robot teleoperation. Each supervised learning on three simulated manipulation tasks, producing 24 runs and approximately 200,000 recorded transitions. Participants watched camera views and could take control using a SpaceMouse controller.

To analyze the recordings, the researchers trained reference policies and value functions—estimates of how promising particular states and actions were for completing each task. Under these estimates, recent progress distinguished intervention moments better than the quality of an isolated robot action. Takeovers tended to follow plateaus or declines in estimated progress.

Human corrections were also imperfect. Some intervention segments moved away from the goal according to the reference value estimates, and corrective actions could differ from the successful strategies eventually learned by the robot. These findings suggest that a helpful takeover need not be an ideal demonstration to imitate indefinitely.

Simulating a more realistic supervisor

The researchers built a progress-aware intervention model that looks back over a short history. It is more likely to take over when estimated progress falls below a threshold. A partially trained robot policy supplies the corrective actions, representing an imperfect supervisor rather than a flawless expert.

In the reported comparison, this model reproduced the ordering of six learning methods observed with human supervisors, giving a Spearman rank correlation of 1.0. That is agreement about which algorithms ranked above others—not proof that the model reproduces human behavior generally or predicts identical success rates.

Its prediction of individual interventions was also far from perfect. On held-out human data, the model had precision of 0.175 and recall of approximately 0.606. It outperformed the alternatives tested, but many predicted takeovers did not correspond to observed interventions.

Guidance without rewarding the mistake

Q-OIL adds a positive learning bonus to transitions where a human supplies the action. This makes the corrected behavior attractive for further exploration. It also retains a relatively small imitation term, so the method does not discard direct learning from corrections altogether.

The central design choice is to separate two value estimators, often called critics. One learns from task rewards alone. The other includes the intervention bonus and guides policy improvement, but uses the task-only critic to estimate future value.

Why separate them? With a single bonus-bearing critic, reinforcement learning can propagate the extra reward backward through earlier actions. That risks making the unsuccessful behavior preceding a rescue look valuable. The two-critic design blocks this particular route of bonus propagation, although shared neural-network representations can still spread its influence to similar situations.

In simulation tests, removing this separation made performance deteriorate as the bonus increased. The result supports the authors’ explanation for why simply rewarding intervention transitions is insufficient.

Results on physical robots

The team tested Q-OIL in simulation, with people supervising simulated robots, and on physical manipulation tasks including placing a pen in a bowl, inserting a peg and unbuttoning a shirt. They report faster learning, higher success and fewer intervention steps than the comparison methods in these experiments.

They also tested two-arm tasks using a pretrained vision-language-action model. After initial demonstration-based fine-tuning, that base model remained frozen; online learning trained a residual policy that adjusted its actions.

For bottle insertion into a backpack pocket, Q-OIL achieved 50% full-task success versus 0% for the behavior-cloning-plus-reinforcement-learning baseline. For pouring coffee beans from a test tube into a beaker, the corresponding results were 70% and 30%. Coffee-task success required at least half the beans to reach the beaker. Evaluation comprised 30 episodes for bag assembly and only 10 for coffee pouring.

Promising, but bounded evidence

These are reported preprint results, not evidence of reliable operation across arbitrary settings. The initial behavioral study involved eight experienced participants in simulation, the later simulated algorithm comparison involved four participants, and the single-arm hardware experiments used one supervisor. Training also began with demonstrations rather than from scratch.

The evidence is therefore strongest for the tested tabletop tasks. The broader idea is nevertheless clear: human corrections can identify useful directions for learning without being treated as perfect answers.