ToolDex records human tool improvisation—and tests where AI falls short

Recording the hand, tool and target

ToolDex focuses on the relationship between a person’s hand, the tool it moves and the target that the tool changes. Examples include crushing nuts, collecting sand and manipulating dough. The aim is to capture not just which object someone chooses, but how its movement affects another material.

The researchers report 10,853 trials covering 60 high-level tool–action–target combinations and 168 variants, including changes in tool instances and task attributes. The dataset includes 45 tool objects and 16 action types. Its unconventional tool inventory includes books and cups alongside familiar implements such as spatulas, ladles and pestles.

The study is an arXiv preprint submitted on November 6, 2026. This report draws on the supplied full paper and appendix, not just its abstract. The supplied material does not establish peer-reviewed validation.

How the demonstrations were captured

Participants performed scripted tabletop tasks inside a calibrated camera rig. Eight external cameras recorded color and depth at 1280 × 720 resolution and 30 frames per second. A headset-mounted stereo camera supplied two first-person color views. The external streams were synchronized in software.

For each task combination, the team varied starting positions and placed four or five distracting tools nearby. This creates scenes in which a model must locate the relevant implement rather than assume it occupies a fixed position.

An automated annotation pipeline estimated tool position and orientation, reconstructed hand poses and outlined objects in video frames. It combined estimates across cameras and refined hand–tool contact to discourage physically impossible overlap. The authors say annotations were manually checked for every trajectory, with additional prompts supplied when automatic tracking or segmentation failed. They nevertheless acknowledge residual noise from occlusion and difficult contacts.

Choosing a tool is not executing a task

The vision-language benchmarks separate tool selection, localization and movement planning. Tool selection uses a permissive rule: an answer counts as correct if any proposed tool matches the demonstrated tool. Consequently, the score does not establish that every suggested alternative would work.

The paper reports that three evaluated models reached 80% tool-selection accuracy. Localization varied considerably, illustrating that recognizing a plausible implement and precisely locating it are different abilities. Bounding-box overlap was measured only for tool predictions that matched semantically.

For planning, models received an initial overhead image and an instruction, then generated two-dimensional movement waypoints. These were compared with paths derived from tracked demonstrations, using both the tool’s center and a point on its surface associated with the prediction’s starting location.

The authors report an 89.4% evaluation-success rate for the model labeled GPT-6-Astra. That measures successfully evaluated outputs, not successful physical manipulation. Gemini Robotics ER had the lowest reported path errors among successfully evaluated outputs. Because failed outputs are excluded from distance calculations, those errors cannot alone establish overall superiority. The paper also gives inconsistent planning sample counts: 180 trajectories in the setup and 160 in the results-table caption.

Better-looking reconstructions can be less faithful

A second benchmark reconstructs static scenes and sequences over time using Gaussian splatting, a scene-rendering method. Researchers withheld one camera to compare generated images with actual recordings. The principal held-out tests covered three static and three dynamic scenes; a separate dynamic evaluation covered 20 scenes but compared against camera views used during training.

For dynamic scenes, adding diffusion-based image repair raised MUSIQ, a learned visual-quality score, from 51.00 to 69.92. Meanwhile, held-out PSNR, which measures pixel-level agreement, fell from 19.60 to 15.97 decibels. These scores came from separate all-camera and held-out-camera runs. The result illustrates why a visually appealing reconstruction should not automatically be treated as a more accurate record.

Robot results remain simulation benchmarks

The robot-learning experiments transferred human movements to robot configurations and compared direct motion transfer, imitation learning and reinforcement-learning approaches. Across six tasks, reported average success was 14.0% for direct retargeting, 4.6% for behavior cloning, 40.5% for ManipTrans and 55.1% for Dex-X.

These are simulation-based results, not demonstrations of equivalent real-world robot performance. Success meant completing a reference interaction without violating task-specific tracking and stability constraints. Moreover, the two reinforcement-learning approaches used different robot embodiments, complicating attribution of their performance gap to learning methods alone.

ToolDex supplies a resource for investigating contact-rich manipulation, but its evidence remains bounded by scripted indoor tabletop tasks, incomplete material coverage and imperfect annotations. The benchmarks identify research challenges; they do not establish general-purpose robotic tool use.