A Surgical Tool Blocks the View. Can Earlier Frames Fill the Gap Without Guessing?

The problem with a convincing guess

An instrument crossing a high-magnification surgical image temporarily hides tissue. That missing view matters not only to a person watching the screen, but also to software that tracks tissue or registers images. Generative inpainting can make a hole look complete, yet the replacement may depict content that was never recorded. Many video inpainting systems also rely on future frames, which rules them out for an online system that must act on the current video.

The team behind TransGHOST asked whether a system could recover some of that lost context using only past video, without inventing image content—and do so quickly enough for a live feed.

Borrowing from the recent past

TransGHOST first segments instruments in each frame. It then searches a short buffer of earlier frames for one where the region now covered is visible. The system aligns that reference with the current image using geometry fitted to visible areas, then resamples pixels from the recorded frame into the instrument mask.

That design sets a clear boundary: in safe mode, reconstructed values come from recorded image content, not a generative model. Each replaced pixel carries information about its source frame and location. If the reference does not provide adequate support, the system leaves the current pixel unchanged and marks it as an abstention. It also preserves pixels outside the predicted restoration mask.

The evaluation used a newly assembled corpus of 13 high-definition recordings of canine cataract surgery. The researchers annotated anatomy and instruments, then created controlled, semi-synthetic tests by combining real surgical imagery and instrument cutouts with images that provide a known unobstructed target. Crucially, they checked that the supplied reference was different from the image used as the scoring target, so simply copying the answer could not pass as reconstruction.

Rather than score the whole image—which would mostly measure unchanged pixels—the researchers evaluated the occluded region. They also tested a pre-declared held-out cohort using instrument cutouts excluded from segmentation training, and measured processing time on an RTX 5000 Ada GPU.

What the tests found

In the deployed configuration, TransGHOST reached 31.89 dB masked PSNR and 0.915 masked SSIM on the validation evaluation. These are measures of pixel-level and structural similarity within the covered region, not proof that the result would help a surgeon make a better decision. Median end-to-end latency was 51.7 milliseconds; the 95th-percentile latency was 85.5 milliseconds. On the held-out cohort, it reached 29.83 dB masked PSNR, with a 95th-percentile latency of 86.1 milliseconds.

The authors also compared restoration methods when masks were matched. On validation samples, TransGHOST’s reference-guided geometric method scored 3.45 dB higher in masked PSNR than LaMa, a generative, reference-free inpainter. That comparison concerns image fidelity under the study’s test conditions; it does not show that one method is safer or more useful in surgery. Their central distinction is that TransGHOST’s safe-mode output has pixel-level provenance by construction.

Reference choice mattered when candidate frames could themselves be obstructed. In that controlled benchmark, choosing a reference based on whether it left the target region visible improved masked PSNR by 3.60 dB over simply choosing the most recent frame. When no available reference supported all of the target region, the system abstained rather than claiming to recover it. Accuracy also declined as references got older: average masked PSNR fell from 33.42 dB at one frame old to 28.06 dB at twelve frames old, about 0.4 seconds at 30 frames per second.

What remains uncertain

These are controlled reconstruction results, not a clinical trial. The authors say fidelity cannot be measured on unmodified continuous video because the true tissue view behind an instrument is not recorded in the same frame. Semi-synthetic tests provide known targets, but cannot establish how the system performs across all real operative conditions. The dataset is veterinary, limited in recordings and operative steps, and the held-out cohort spans three videos. The study did not test integration with a surgical robot or whether restored pixels improve tracking, guidance, or surgical outcomes.

The latency results are also hardware- and pipeline-specific: the 95th percentile stayed below 100 milliseconds, but the 99th percentile was 101.4 milliseconds on validation. And provenance does not guarantee correctness: a pixel can be traceable to an earlier frame yet still be misaligned or visually inaccurate.

The manuscript is an arXiv preprint submitted to Elsevier, not evidence here of peer-reviewed clinical validation. Its authors say code and model architecture are intended for release upon acceptance; the dataset is not slated for public release, though access may be requested under institutional and data-use restrictions. For now, TransGHOST is best understood as a tested approach to traceable, short-term reconstruction—not a proven surgical aid.