OmniPiano tests AI control with up to five virtual piano-playing hands

A keyboard as a control challenge

Playing a piano requires more than reaching the right location. Fingers must press, hold and release particular keys at particular times, often while other fingers are doing something different. OmniPiano turns those demands into a collection of tests for reinforcement learning, in which a controller learns through interaction with an environment and feedback from rewards.

The benchmark, presented by Fan Xu and colleagues, builds on RoboPianist and the MuJoCo physics simulator. It supports one to five simulated Shadow Hands, with up to 111 continuous action dimensions—the numerical commands governing the hands and sustain pedal. Musical targets come from MIDI scores, which specify the required notes and timing.

The authors describe 912 task settings across four tracks and support for 36 reinforcement-learning baselines, alongside four language-model-based agents. The contribution is a shared testing framework, rather than a new algorithm that solves robotic piano playing.

Four ways to test the same skill

The standard track examines playing performance across different hand configurations and reward designs. Some configurations let hands move freely across the keyboard; others confine each hand to an assigned region. Rewards encourage correct key presses and fingertip placement while discouraging excessive actuation. One reward option assigns fingertips to target keys dynamically, avoiding the need for prescribed fingering annotations.

The robustness track introduces controlled disturbances. These can corrupt observations, actions or reward signals, or change simulated gravity, fingertip friction and starting hand positions. Researchers can test disturbances individually or combine them.

The safety track adds explicit costs for behaviors such as exceeding joint ranges, using too much actuator power or generating unwanted contact between hands. Another constraint limits actuation of a designated thumb to represent restricted use of an injured finger. These are simulation-defined constraints, with thresholds chosen as benchmark parameters—not demonstrated guarantees of physical safety.

Finally, the multi-agent track divides control among several cooperating agents. It varies how many controllers share the hands, how much their keyboard workspaces overlap, how evenly responsibilities are distributed and what each controller can observe. Safety and multi-agent learning remain separate task families; the benchmark does not yet combine them into a safe multi-agent setting.

Extra hands can create extra difficulty

In the reported standard experiments, adding hands did not consistently improve note accuracy. Restricting hands to predefined workspaces generally produced faster, more stable learning than allowing unrestricted movement. The authors interpret this as evidence that additional freedom can also make exploration and coordination harder.

The study measures key-activation accuracy using F1, which balances pressing incorrect keys against missing required ones. Its survey of musical pieces suggests that note density is more informative about difficulty than duration alone, although fingering and spatial transitions also matter.

The language-model agents approached the task differently from reinforcement-learning policies. They wrote Python controllers that generated open-loop movement trajectories, then revised those controllers using simulator feedback. Each received ten optimization attempts, and the researchers reported the controller with the best F1.

The authors report that language-model agents could improve within a few attempts and sometimes outperform reinforcement-learning baselines on a challenging five-hand piece. However, their results varied more and required substantial token consumption. Reinforcement-learning methods, evaluated over five million environment steps in the two-hand comparison, improved more steadily. These different protocols do not establish a simple, like-for-like efficiency ranking.

Disturbances, constraints and cooperation

Action disturbances and changes to initial hand position caused the largest performance losses in the robustness experiments. Gravity and friction changes were generally less disruptive under the tested conditions.

Safety results depended on the constraint. Some methods reduced joint-range violations without losing playing performance relative to unconstrained PPO. Power constraints, by contrast, reduced both violations and musical performance. In a hand-count comparison with a fixed aggregate power allowance, four-hand policies struggled more than two- or three-hand policies—a result that does not isolate control dimensionality alone.

Cooperation offered a different outcome. On a fixed four-hand task, distributing control among more agents improved performance in the reported comparisons. That means more controllers, not more hands. Overlapping workspaces generally made coordination harder, while broader observations often helped.

The authors explicitly limit their conclusions to simulated piano playing. Results may depend on the hand model, contact dynamics and chosen disturbances or constraints, and may not transfer directly to physical robots or other manipulation tasks. OmniPiano therefore offers a structured way to expose control weaknesses, rather than evidence that those weaknesses have been overcome.