TouchéWorld combines camera navigation and hand control in AI-generated scenes

A generated courtyard becomes more interactive if a visitor can approach a tree, reach into its branches and then continue walking. That is the kind of experience targeted by TouchéWorld, a framework that combines camera-guided exploration with hand-controlled interaction in generated video.

In a November 4 arXiv preprint, Jiaao Yu, Yanjian Zhang, Haoyang Jiang, Guillaume Wisniewski, Nadi Tomeh and Shijie Jia describe a system that starts from an image and a scene-description prompt. Camera controls specify the viewpoint trajectory, while hand controls specify poses and movement. The authors report experiments showing action-dependent continuations, alongside substantial trade-offs between visual quality, control accuracy and few-step generation.

Two kinds of control, one video

TouchéWorld builds on the LTX-2.5 video-generation backbone with a SolarWM camera adapter. Rather than retraining the entire system, the researchers keep the existing backbone and camera adapter frozen during joint-control adaptation and train additional hand-conditioning components and low-rank adapters, compact sets of adjustable parameters.

The hand input has two complementary representations. Rendered skeleton sequences indicate where the hand and its joints should appear. Numerical features describe wrist translation, joint geometry and whether the hand annotations are valid. Together, these signals provide spatial guidance and information about movement over time.

Training uses 40,958 first-person video clips, mainly from EgoLife and EgoStandard. Because the source videos do not all contain complete camera and hand annotations, the team estimates missing information, filters inconsistent estimates and synchronizes the controls with the frames. This alignment matters: a viewpoint change should not be mistaken for an object moving because a hand pushed it.

Responding without knowing future actions

The researchers first train a bidirectional model, called the teacher, which can use information across a sequence. They then adapt it to generate successive chunks without access to controls from future chunks. Here, “causal” describes the model’s information access; it does not establish that the system has learned physical cause and effect.

A further distillation stage trains a student to generate each chunk using four denoising steps. During this training, the student learns from its own generated history, rather than relying exclusively on recorded preceding frames.

The authors describe an interface that accepts updated controls while generation proceeds and applies them to subsequent chunks. Already committed frames remain unchanged. This supports a feedback loop, but the report provides no measured latency or hardware throughput establishing real-time responsiveness.

What the experiments found

The team introduces TouchéBench to separate four questions: whether hands follow their controls, whether camera and hand movements work together, whether new actions produce different continuations, and whether objects respond appropriately.

On 200 held-out HOI4D clips, the teacher achieved 65.8% hand-keypoint accuracy at the reported normalized-distance threshold, compared with 61.2% for Hand2World and 59.1% for JCV. Missing hand detections counted as incorrect. Accuracy fell to 57.9% for TouchéWorld’s causal model and 54.0% for its four-step version.

For action-dependent continuations, eight scenarios each supplied three alternative hand-action branches after a shared context. The authors report overall checklist pass rates of 75.0% for the causal model, 71.5% for the teacher and 68.8% for the four-step model, versus 51.4% for JCV and 33.3% for Hand2World. These are averages across evaluation criteria, not percentages of fully successful interactions.

The harder criteria expose remaining weaknesses: the causal model scored 55.6% on judgments covering action execution, intended final state and exclusion of alternative outcomes. Object measurements also gave mixed rankings. JCV achieved the highest object-mask overlap, while TouchéWorld’s causal model had the lowest average object-center offset. Neither measurement alone establishes successful manipulation.

Promising interaction, limited evidence

The coordination test contains only ten purpose-recorded clips, and the branching test contains 24 continuations. Both use automated judgments from a model identified in the report as GPT-6 Astra, accessed through Codex, rather than a reported human evaluation. Comparisons also involve different supported inputs: JCV receives hand conditioning but lacks a camera-control interface. Teacher and causal models use different context initialization procedures in the branching evaluation, further complicating direct comparisons.

The supplied extraction also contains inconsistent aggregate and component scores in one detailed interaction table, warranting caution about fine-grained interpretation.

The preprint reports experimental results, not guarantees of stable physics or persistent world state. Its clearest contribution is a framework for studying navigation and hand interaction together—and evidence that reducing generation steps can weaken control even when some image-quality measures improve. Peer-review acceptance is not indicated in the supplied record.