IdealCache plans when image and video AI should reuse its calculations

Diffusion transformers generate images and videos through a sequence of denoising updates. Each update can require an expensive pass through the neural network. Feature caching reduces that work by saving intermediate calculations and reusing them at later steps. The challenge is deciding when those saved calculations have become too stale.

In an arXiv preprint, Shang Xue, Li Zhu and Ping Chen propose IdealCache, a method that plans when to refresh the cache without retraining the underlying model. Their central question is whether a schedule should imitate every stage of a standard generation run—or instead be selected using a reference path toward a target result.

What an “ideal trajectory” means

The method operates in latent space: the numerical representation in which the model processes an image or video. For a candidate target output, the researchers construct a straight line between that output’s latent representation and the starting noise. They call this an ideal trajectory.

“Ideal” does not mean a verified perfect image, nor does it mean the model naturally follows a straight path. The line is a measuring reference. For text-to-image and text-to-video generation, the target is itself an output from the standard 50-step pipeline. For image editing, the source image supplies a reference that encourages preservation of existing content.

At each candidate refresh point, IdealCache runs the full transformer and saves residuals—the changes its blocks add to their inputs. It then tests how reusing those residuals across subsequent updates moves the latent representation away from the reference line. The squared distance at the interval’s endpoint becomes that interval’s cache-deviation score.

The researchers turn candidate timesteps into nodes in a directed graph, with interval scores as connection costs. They average costs across reference runs, then use dynamic programming—a procedure that combines solutions to smaller subproblems—to find the lowest-cost schedule for a fixed number of full transformer evaluations. During generation, only the selected steps refresh the cache.

The search is exact for this graph objective. That is narrower than proving that it finds the best possible final image or video. The paper’s mathematical argument for independent interval errors assumes a linear reference flow and constant cached velocity within an interval; actual generation can accumulate errors as it departs from reference states.

What the experiments found

The experiments used one NVIDIA H800 80GB GPU and compared against standard 50-step runs. Reported wall-clock speedups include generation-pipeline components such as the text encoder and image decoder, not just transformer arithmetic. Offline schedule construction is a separate cost.

On Qwen-Image with DrawBench prompts, seven full transformer evaluations reduced latency from 46.54 to 7.33 seconds: a 6.35-fold speedup. However, ImageReward, a learned score intended to reflect human image preferences, fell from 1.2092 to 1.1068, an 8.47% decrease. With nine evaluations, the speedup was 5.18-fold and the ImageReward decrease was smaller, at 2.30%.

Image editing produced some scores above the unmodified pipeline. On the English portion of GEdit-Bench, Qwen-Image-Edit with seven full evaluations achieved a 5.84-fold speedup and an overall score of 7.59, compared with 7.54 for the baseline. On FlowEdit, background-preservation scores increased from 0.665 to 0.687 with 25 evaluations, although ImageReward was slightly lower. These results support improvements on particular measures, not universal superiority.

For HunyuanVideo, tests generated 61-frame videos at 320 by 512 resolution. Ten full evaluations reduced latency from 80.94 to 18.74 seconds, a 4.32-fold speedup. Evaluation included SSIM, which measures structural similarity, and LPIPS, which measures perceptual differences through learned visual features. These quantify resemblance to reference outputs rather than establishing every aspect of video quality.

Planning costs and remaining limits

The main image experiments built average graphs from 1,000 reference runs; video used 50. A smaller experiment transferred schedules built from one prompt to other inputs, with relatively small score differences on image generation and editing. Constructing a single-prompt, 50-node graph reportedly took about 80 seconds. Reusing a schedule spreads that planning cost across subsequent outputs; it does not eliminate it.

The evidence remains limited to one base model per task. Complex video may still require multiple reference runs, and very small evaluation budgets degrade quality. Generation targets also inherit failures of the original model: planning toward a flawed reference does not fix its content. The supplied work is an arXiv preprint, and its reported benchmark results should be read as evidence for a promising scheduling approach, not a guarantee of lossless acceleration.