When searching harder makes a robot worse

A good judge of examples can be a bad guide

A simulated robot arm already knows how to open a drawer. Give it a planner that searches for improvements using a reward learned from human comparisons, and it can stop opening the drawer altogether.

That is the sharpest result in a preprint by Soovadeep Bakshi and Apurba Bose. Their study examines a planning version of Goodhart's law: optimizing a measure can expose ways in which it differs from the objective it supposedly represents.

The robot need not learn anything new during deployment. Searching harder against an already-trained reward model can be enough to make performance worse.

How the researchers separated judging from planning

The experiments used simulated walking and manipulation tasks, including HalfCheetah, Hopper, drawer opening, plate sliding and window opening. Reward models learned from comparisons between short behavior segments. Walking-task comparisons came from a crowd-sourced dataset; manipulation comparisons included synthetic judgments and blind judgments made by the two authors.

A baseline controller was trained to imitate recorded behavior. Around its proposed movements, a Model Predictive Path Integral controller, or MPPI, generated candidate action sequences. Typically, it evaluated 256 sequences extending 20 steps ahead, weighted them using their learned rewards, and executed the first action of their weighted average. It repeated this process after each step.

A lower temperature made the planner concentrate more strongly on its highest-scoring candidates. Crucially, planning used the simulator's actual dynamics, not a learned prediction model. This removed one possible explanation for failure: inaccurate predictions of how the simulated world moves.

The researchers measured performance using the environment's reward, which they call true return, and counted task successes for the arm. This benchmark reward is distinct from both the learned score and a person's judgment.

The drawer that stayed shut

With a reward model trained on 300 human-labeled drawer comparisons, gentle search opened the drawer in all 20 evaluation runs. At temperatures of 0.1 or below, it opened in none. The imitation controller, executed directly without planning, succeeded in all ten baseline runs.

Yet the human-label reward model achieved 97% accuracy on held-out comparisons. The person supplying those training labels agreed with the benchmark ranking on 97.6% of comparisons where they expressed a preference.

Those strong numbers concealed a critical weakness. Some clips looked indistinguishable to the person, although their benchmark returns differed. Controls implicated these difficult comparisons in the planning failure. In recorded episodes, the human-trained scorer guided the arm into pulling in front of the handle without touching it.

The outcome was not universal. Human labels resisted exploitation better than synthetic labels on plate sliding. More training comparisons made drawer planning beneficial in one setting, and harder search helped on window opening. Hopper also lost performance under hard search with a perfect scorer, showing that not every loss was reward-model exploitation.

Measure errors where search actually happens

Held-out accuracy asks whether a model predicts comparisons resembling its labeled examples. A planner asks a different question: which of its own candidate futures deserves the most weight?

The study found that these checks could disagree sharply. Across the tested weighting rules, performance approximately followed how far search shifted weight away from the original proposals. This was an empirical pattern within tasks, not a universal law.

The authors also tested simply choosing the highest-scoring action from many candidates. With verifiers trained on 500 synthetic comparisons, selecting among more actions eventually performed worse than taking one randomly perturbed action. With 2,000 comparisons, selection helped instead.

A brake with important conditions

The theoretical analysis bounds how much selection can lose through uneven scoring errors across candidate plans. Under that error model, a chi-squared-regularized weighting rule with an appropriately chosen temperature gives the best worst-case selection guarantee. If uncertainty is too large, the rule declines to search.

This does not guarantee that an entire robot episode improves. Averaging actions, adding proposal noise and repeatedly executing only the beginning of a short plan introduce other effects. The theorem controls the selection component, not all those consequences.

Calibrating the rule with 100 candidate-plan comparisons ranked by benchmark return produced the safest tested results. Across five independent calibration sets drawn from starts excluded from evaluation, it succeeded in every drawer and plate-slide run, retaining roughly 86–94% of the direct imitation controller's return on average. Caution also sacrificed gains where search was useful.

Who checks the checker?

Benchmark-ranked calibration is an oracle that deployed robots generally lack. Human calibration introduced another problem: someone whose labels trained the scorer could share its bias and endorse unsafe search. A second evaluator, a correction for indistinguishable pairs, or a more conservative estimate reduced the damage without establishing a general solution.

These are simulation results with small state-based networks, limited runs and only two author-labelers for the manipulation tasks. Their practical warning is narrower than “planning fails”: validation should examine the choices optimization actually creates, and confidence borrowed from familiar examples may not survive harder search.