Can image generators replace attention with multiplication?

Rethinking the machinery of denoising

An image generator can begin with random noise and gradually transform it into a recognizable picture. The network guiding that process must estimate what a clean image could look like at each stage. But does attention—the information-mixing mechanism central to Transformers—provide the most suitable machinery for this job?

In a November 2026 preprint, Siddharth Roheda and colleagues at Samsung Research Institute Bangalore propose an alternative called Polynomial Denoisers, or PoD. It works directly with pixels and replaces attention and conventional feed-forward sublayers with learned polynomial interactions. Their evidence combines mathematical arguments, a simplified geometric experiment and image-generation benchmarks. These support a promising architectural alternative, not a general verdict against Transformers.

Why polynomials enter the picture

The starting hypothesis is that natural images occupy a comparatively small, structured region within the enormous space of possible pixel combinations. Mathematicians describe such a region as a low-dimensional manifold. A patch containing an edge, for example, has structure that arbitrary random pixels lack.

Under the paper’s assumptions, the best squared-error denoiser is the conditional mean: an average of possible clean signals, weighted by how well each explains the noisy observation. For Gaussian noise, the authors express this denoiser locally through a Wiener–Volterra series, involving progressively higher-order products of input values.

That local expansion need not converge across all relevant inputs. The authors therefore give a separate approximation argument: a polynomial acting on a compressed representation can approximate the denoiser on an assumed compact, smooth patch manifold scaled by the signal’s strength at a fixed sampling time. In Proposition 1, the approximation exponent depends on the manifold’s intrinsic dimension rather than the original number of pixel coordinates.

The location of that guarantee matters: it bounds error on the scaled clean-data manifold, not on typical noisy inputs, which lie off it. Extending the bound to a surrounding tube requires additional assumptions, including a smooth data density bounded away from zero, a tube narrow enough to have unique nearest points on the manifold, and a further restriction involving the projected manifold’s geometry. Noise spread over many pixel coordinates can put typical inputs outside that tube, especially before the low-noise end of sampling.

Even where the extension applies, compression adds error. The paper’s worst-case squared compression-error term can scale as O((D − d_M)σ²/t²), where D is the number of pixel coordinates per patch, d_M is the manifold dimension, t is sampling time and σ = 1 − t is the noise level. This term vanishes late in sampling at fixed D, but retains ambient-dimension dependence. The intrinsic-dimension approximation exponent therefore does not establish an ambient-dimension-independent total denoising error across the sampling schedule.

This is an existence result under geometric assumptions, not a demonstration that training will find the required polynomial efficiently. Real images need not lie exactly on a smooth manifold, and the study does not measure the intrinsic dimension needed to validate its chosen bottlenecks.

Multiplication instead of attention

PoD divides an image into patches and compresses each patch through a narrow linear embedding. Its Volterra mixers then project their inputs, multiply pairs of projected values and combine the products. One mixer connects patch positions; another combines features within each patch. Factoring these interactions avoids storing a huge table of every possible quadratic coefficient.

Multiplication already supplies nonlinearity, so the image-processing residual path needs no conventional activation such as GELU. However, the implemented network is not entirely polynomial: it retains RMS normalization and a separate conditioning network for diffusion time and image class.

That distinction matters. The exact polynomial-representation theorem excludes normalization, and the appendix explicitly does not establish the same containment for the normalized network actually trained.

The paper also shows that a single softmax-attention layer cannot exactly equal a non-affine polynomial on an open input region. A broader result applies to globally polynomial mixing rules that form nonnegative, normalized averages of affine values. Neither establishes that attention cannot approximate a denoiser. The authors acknowledge that Transformers can do so; a quantitative approximation-rate disadvantage at matched budgets remains a conjecture.

What the experiments report

A controlled experiment embeds a one-dimensional spiral in spaces with 2, 32 or 512 coordinates. Predicting the clean signal preserves the spiral more successfully as dimensionality increases than directly predicting noise or velocity. Under clean-signal prediction, the Volterra model has the lowest reported average distance to the spiral. This tests a known geometry, not the full complexity of photographs.

For the main experiments, the researchers train class-conditioned generators on ImageNet at 256-by-256 and 512-by-512 resolution. The models predict clean images, while training measures a derived velocity error. They use no pretrained tokenizer, perceptual-loss network or representation-alignment model.

On 50,000 generated samples at 256-by-256 resolution, the largest PoD model reports a Fréchet Inception Distance, or FID, of 1.48 versus 1.82 for the corresponding JiT baseline. Lower FID indicates closer agreement between generated and reference image distributions in an image-feature representation; it is not a percentage accuracy or a direct human-quality rating.

The budgets are comparable, not identical: PoD uses about 1.6 billion parameters and 319 billion floating-point operations per forward pass, versus roughly 2 billion and 383 billion for JiT. At 512-by-512, the corresponding FIDs are 1.55 and 1.78. PoD improves FID at every reported scale, but not every Inception Score.

A useful alternative, not a universal winner

Other reported systems still achieve lower FID, sometimes with more computation or pretrained components. The sample-selection description also says the illustration grids were curated, despite captions calling them uncurated.

The strongest conclusion is narrower than the theoretical framing: multiplication-based mixers can support competitive pixel-space generation without attention or pretrained components. Establishing that their mathematical structure explains the gains—and guarantees superior efficiency—requires evidence beyond these reported benchmarks.