A small pixel-level touch-up sharpens image generation

Many image generators first compress pictures into a learned representation, then generate within that compressed space. This can make computation easier, but it also makes the generator depend on a separate compression model. Pixel-space diffusion models take another route: they work directly with image pixels. The trade-off is that handling detail can be difficult, especially when a transformer represents an image as a grid of relatively large patches.

The study focuses on Just Image Transformer, or JiT, which predicts the clean image rather than the noise used to create it. The authors describe a possible weakness in its design: each large patch is decoded separately, so the model must coordinate fine details and boundaries between patches using patch-level features. Their proposed fix is a small convolutional “pixel pathway.” It processes smaller regions across the image, can use the current noisy image directly, and is intended to help nearby details line up across the original patch boundaries.

Refinement when detail becomes clearer

The central question is whether this added pathway must run throughout denoising. Early in generation, an image is very noisy and many possible final pictures remain plausible. The authors reason that fine detail is hard to settle at that point; it becomes more meaningful as the image takes shape. They therefore test a two-head arrangement: a lightweight, overlapping linear output head handles earlier steps, and the convolutional head takes over later. They also test gradually blending between the heads.

The team starts from pretrained JiT models and fine-tunes them in two stages. In the reported setup, the new pathway is initially trained while the transformer backbone is held fixed, then both are fine-tuned together. The second stage changes how denoising timesteps are sampled and how training errors are weighted, giving more attention to the near-clean end of the process. This matters when interpreting the results: the comparison is not simply “add convolutions and change nothing else.” The training and sampling procedures are also examined and adjusted.

For evaluation, the researchers generate class-conditioned images at 256-by-256 resolution on ImageNet. They report FID, a commonly used measure comparing statistical features of generated images with those of real images; lower values indicate a closer match under that measure. Their evaluation uses 50,000 generated images for reported headline comparisons, with settings selected through sweeps of guidance and model-weight averaging.

What the experiments show

In smaller JiT-B and JiT-L experiments, adding the convolutional head improved FID relative to the corresponding pretrained models in the paper’s comparisons. The authors also report that running the convolutional pathway only late in denoising produced little or no quality loss compared with using it throughout, while reducing the added computation. The precise savings depend on model size and head schedule; the transformer backbone still runs at every denoising step.

For the larger JiT-H/16 model, the paper reports a gFID of 1.78 for its dual-head system, compared with 2.34 for its reproduced JiT-H/16 baseline using the corrected sampling procedure. The study also reports an earlier JiT result of 1.86, illustrating why the choice of reproduction and sampling settings matters. The authors say the improvement was achieved without adding extra losses such as representation-alignment loss. These are results on a particular ImageNet setup, not evidence that the method will improve every image generator or every kind of image task.

The timing of the switch matters. In an ablation on JiT-B, activating the convolutional head at the final 20% of denoising gave a gFID of 3.42, close to 3.39 for a gradual transition over an earlier interval. Delaying activation until very near the endpoint performed worse in that test. Thus, “late” does not mean “at the last instant”: the refinement head appears to need enough steps to contribute.

What remains uncertain

FID is a statistical benchmark, not a complete measure of whether images look realistic, contain correct details, or suit a particular use. The paper’s experiments are centered on ImageNet at one resolution and on JiT models; they do not establish performance for text-to-image systems, other resolutions, or other architectures. The authors also note that they did not determine the best point during training to introduce the pathway, and that their exploration of alternative head designs was limited. Since the backbone remains active, late refinement reduces the pathway’s extra cost rather than eliminating the main generation cost.

This manuscript says it was originally submitted to ICLR 2027 and is available as an arXiv preprint. The supplied source does not establish that it has completed peer review. Its findings should therefore be read as promising experimental evidence, not as settled consensus. The paper also says code and checkpoints are intended for release; independent reproduction would help clarify how robust the gains are across settings.

Source: arXiv:2611.13120v1.