A robot reads the recipe—and finds its ingredients

A person can glance at a printed recipe, recognize a cup by its label, and adjust a pour depending on whether the cup is full. Lab robots have generally been built for a more orderly world: their procedures are translated into scripts, and containers are put in preset locations or tagged so a camera can identify them. Curie’s Jarvis, described in an arXiv preprint, tests a different approach: could a robot arm manage simple chemistry on a bench arranged for people?

The system combines cameras, a robot arm and language-model software. It starts with a photo of the printed instruction sheet. A vision-language model extracts the steps, and a planner turns them into calls to a fixed set of manipulation skills, such as scooping, pouring, stirring and using a pipette. The model does not control every movement directly: the skills handle the physical motions and checks, while the model selects tasks and some parameters.

Four cameras look at the bench from different angles. Software identifies objects and estimates their positions from the views. To distinguish containers that look alike, the system reads the paper labels beneath them and combines readings across cameras by voting. Before most actions, it locates the target again, rather than assuming the container is still where it was earlier. It also varies movement speed according to what the robot is carrying—for example, moving more cautiously with liquid than with nothing in its gripper.

The researchers tested the setup on a Franka Panda robot arm using two protocols from a children’s chemistry kit. One, called Magic Beaker, combines liquids and powders to produce a fizzing, purple mixture. The other, a jellyfish demonstration, uses indicator drops that sink through oil and turn a solution blue. Each protocol was run five times. The robot completed all five Magic Beaker runs and four of the five jellyfish runs without human intervention. The failed run ended when the system mistook a scoop for the pipette. The tools were unlabelled, so the label-reading method could not help; the robot had to recognize the pipette from its appearance.

In a separate label-identification test, the system found the correct container in all 160 attempts, using printed labels and handwriting from three people. That perfect result applies to this particular test setup, not to every label, camera angle or lab. The study also examined containers moved away from their labels. In those tests, it sometimes returned “not found” rather than naming the wrong container—an important distinction, but not proof that it can handle arbitrary changes to a busy bench.

The paper’s strongest evidence is a demonstration of feasibility: a robot can connect a photographed protocol to physical actions while relying on ordinary labels and repeated visual checks. But the scope is narrow. The experiments used one robot arm, one bench and two simple kit activities. The reagents were already measured into containers, and the protocols did not depend on exact quantities. The robot did not weigh or otherwise measure what it transferred, so the results say nothing about its ability to deliver precise amounts—a requirement in many research experiments.

Other capabilities remain proposals rather than tested results. Although the system is designed to re-locate objects if a person moves them, the researchers did not test people moving items during a run. Likewise, they describe adaptive movement speeds, but did not separately compare those choices with fixed-speed movement. The end-to-end tests involved only slight shifts between runs, with no objects swapping places. Runs took roughly 17–30 minutes, in part because the cameras were processed one after another.

The work is a preprint: it was posted on arXiv, and the supplied source does not establish that it has undergone peer review. Its findings should therefore be read as an initial engineering demonstration, not as a validated general-purpose chemistry platform. The authors identify next steps including testing human movement during experiments and protocols that require measured quantities. For now, Curie’s Jarvis shows how far a robot can get when it reads the same kind of instructions and labels a person might use—and how much remains to be tested before that flexibility can be trusted in research labs.