Can a One-Step Image Generator Learn Preferences Without Losing the Plot?

A diffusion model usually builds an image through many gradual denoising steps. A one-step generator skips that journey during image generation: it maps noise directly to an image, which can make generation much faster. The trade-off is that a fast model may produce less appealing images or follow a prompt less reliably. This paper asks how preference-based training can improve one-step generators without allowing them to stray too far from the image distribution learned by a reference diffusion model.

The authors focus on a mismatch in earlier training approaches. These methods reward the finished image but compare the generator with the reference model across multiple noisy stages. In the authors’ analysis, that arrangement can give reward too much influence: the generator may concentrate on outputs that score well, even if they are less faithful to the reference model. They call this failure mode “terminal reward domination.” A mathematical example and a one-dimensional toy experiment illustrate the proposed problem; they do not, by themselves, establish how often it occurs in real-world image generation.

The proposed method, Diff-Instruct with Diffused Reward (DIDR), changes where the preference signal enters training. It begins with a standard reinforcement-learning-from-human-feedback idea: balance a reward against a penalty for moving too far from a reference distribution. The resulting ideal target favors images with higher reward while retaining a connection to the reference. DIDR then carries that reward-adjusted target through the diffusion process, defining a corresponding target at each noise level instead of attaching the reward only to the final, clean image.

The paper derives a “Diffused Reward Score” (DRS), a correction to the reference model’s score—the signal that guides denoising. Computing the correction exactly would require an unavailable expectation over possible clean images behind a noisy image. To make training practical, the authors introduce the “Diffused Reward Proxy” (DRP): they run short, differentiable denoising chains with a frozen reference model, score the resulting images, and use those scores to estimate the correction. A separate “Teaching Assistant” score model tracks the one-step generator’s evolving image distribution. The reference model and reward model stay frozen during this training procedure.

The theoretical result concerns an idealized objective: under the paper’s stated conditions, minimizing its trajectory-level divergence has the same distribution-level minimizer as the corresponding clean-image preference objective. The working algorithm is approximate, however. It uses short denoising chains and a finite number of samples, so practical training is not identical to the ideal objective.

For experiments, the researchers trained on text prompts from LAION-Aesthetic-6.25+ without using image training data. They used PickScore as the training reward. Their main tests fine-tuned a one-step SDXL generator; they also applied the approach to Z-Image, a larger diffusion-transformer model. Evaluation used automated preference and text-alignment metrics, plus FID, a measure of how closely generated-image features match those of a reference image set. PickScore was both the training reward and an evaluation metric, so results on other preference measures help show whether gains extend beyond the signal directly optimized.

On SDXL, the standard DIDR model scored 23.5 on PickScore and 18.8 on FID in the paper’s evaluation. The extended-training version scored 23.9 and 20.6, respectively. Higher PickScore is better, while lower FID is better; the change illustrates a preference–fidelity trade-off rather than a universal improvement. The authors report gains over the strongest prior one-step baseline on several preference measures, including ImageReward and HPSv2.1. The standard version also had the highest reported DPG-Bench and GenEval scores among the compared one-step SDXL methods, though its CLIPScore was slightly lower.

On the 6-billion-parameter Z-Image backbone, the DIDR-trained one-step model exceeded the 50-step base model on the paper’s preference metrics. This result is specific to the reported setup and metrics: against the 8-step Z-Image-Turbo model, it led on some measures but trailed on PickScore, HPSv2.1 and GenEval. It does not show that one step is generally better than many, or that the method is uniformly superior across models and tasks.

Important uncertainties remain. Automated preference scores are proxies, not direct evidence that people will prefer images in everyday use; FID measures distributional similarity, not whether an image obeys a particular prompt. The authors warn that backpropagating through a reward model may amplify its biases or encourage reward hacking. DIDR’s training is computationally demanding because each generator update requires multiple differentiable denoising steps. The paper also identifies anatomical errors, incorrect counts, structural distortions and tangled subject–background relationships as continuing failure cases.

The supplied source identifies this work as an arXiv preprint; its review status is not established by the evidence available here. The results are reported by the authors and would benefit from independent replication and broader evaluation. The contribution is a proposed training objective and benchmark results, not proof that speed–quality trade-offs have been solved.