SurgTransfer learns multi-step actions to generate surgical videos in new contexts

SurgTransfer aims to separate what happens in a surgical video from the particular scene in which it happens. Given a source demonstration and a single image of another surgical context, the system generates a video depicting the source actions in that target scene. Saurabh Koju and colleagues describe the approach in an arXiv preprint submitted on November 3, 2026.

The distinction between generated video and physical action is central. Here, “transfer” means reproducing aspects of a demonstrated maneuver in a simulated visual sequence. The paper presents this as an initial step toward transferring surgical expertise, rather than an established clinical capability.

Learning more than the next movement

The authors argue that learning actions only between adjacent frames is too narrow for surgical procedures, whose movements form longer, goal-directed sequences. Neighboring frames can also look very similar, allowing a model to reconstruct the next image without learning much about the broader action.

SurgTransfer instead learns a composite action representation spanning multiple frames. Its latent action model compresses observed changes into continuous numerical representations. During reconstruction, the decoder receives the initial frame and those inferred actions, rather than a succession of real intervening frames. This setup is designed to encourage actions that compose into a longer sequence.

A second component, adapted from Stable Video Diffusion, generates future frames conditioned on a starting image and the learned actions. It produces several frames together, instead of generating every frame separately from earlier predictions. The authors report that this reduces the progressive visual deterioration seen in their autoregressive comparisons, although it does not eliminate deterioration over longer predictions.

The action-learning process uses videos without explicit action labels or robotic movement measurements. However, the evaluation is not entirely annotation-free: the researchers use available surgical-step labels to select comparable source and target segments in one dataset.

Testing transfer between surgical scenes

The study uses SurgToolLoc-2022 and PitVis-2023, with an 80:20 training-to-test split for each. PitVis contains 25 endoscopic pituitary surgery videos averaging more than an hour each. AdaWorld, an existing latent-action world model, serves as the main action-transfer baseline.

For PitVis, source and target segments come from different videos but represent the same surgical step. In the main SurgToolLoc evaluation, each 30-second clip is divided into two temporal segments to create source–target pairs. These comparisons therefore test transfer between procedurally related scenes, not unrestricted transfer between arbitrary operations.

The reported prediction horizon is 14 frames. On Content-Debiased Fréchet Video Distance, a measure that emphasizes motion dynamics more than conventional video distance, lower scores are better. SurgTransfer scored 141.41 against AdaWorld’s 341.76 on SurgToolLoc, and 157.34 against 513.27 on PitVis—reductions of approximately 59% and 69%, respectively.

The researchers also tested action cycle consistency: infer actions from a transferred video, apply them back to the original context, and compare the result with the source. SurgTransfer achieved lower perceptual error on both datasets. These measurements support the authors’ transfer claims, but they measure generated-video behavior rather than successful physical execution.

Reconstruction experiments provided a separate check on video fidelity. SurgTransfer improved several reported measures over AdaWorld, although it did not lead every metric across all tested variants. For example, the optical-flow baseline achieved a lower video-distance score in reconstruction, while SurgTransfer performed better on that measure in the main action-transfer comparison.

What surgeons judged—and what remains limited

Five certified surgeons independently assessed 40 generated videos. SurgTransfer received higher average ratings than AdaWorld across visual plausibility, instrument operation, environment feedback, surgical intent and action transfer. Visual plausibility averaged 3.42 versus 2.65; action transfer averaged 3.15 versus 2.63. Average inter-rater reliability, measured using ICC(2,k), was 0.54 for SurgTransfer and 0.51 for AdaWorld.

The paper also documents failure modes. Both systems deteriorated as predictions lengthened. SurgTransfer showed a quality drop when it restarted generation using its last generated frame as the next context. A supplementary example showed exaggerated instrument movements, which the authors interpret as excessive weighting of the action signal.

The authors limit the framework’s applicability to contexts with similar anatomy or views, the same procedure and the same instrument embodiment. Its predefined action trajectory also prevents a truly interactive experience, and its unsupervised evaluation leaves performance on downstream tasks with ground-truth actions unresolved. The contribution is therefore a method for generating more consistent action-transfer videos within the tested settings, with broader expertise transfer remaining a research objective.