SIFR balances positive and negative feedback to stabilize image-model training

Why rewarding good images is not enough

Image generators based on rectified flow learn how to transform noise into images through a sequence of changes. Their learned velocity field specifies the direction and size of those changes along the generation process.

Reward-based fine-tuning adjusts a pretrained generator using scores assigned to its own outputs. But repeatedly emphasizing highly rewarded examples can concentrate the output distribution around reward-maximizing images. The SIFR preprint, posted on November 6, 2026, identifies another weakness: positive training weights encourage fitting good examples without explicitly discouraging poor ones.

The proposed method, Stable Signed Flow Reinforcement, addresses both problems. It combines positive and negative training signals with a feedback-controlled restriction on changes to the velocity field.

Negative feedback with an adjustable restraint

SIFR replaces raw reward weights with signed advantages: scores expressed relative to the reward mean and normalized by the reward standard deviation. Above-average outputs receive positive weights; below-average outputs receive negative weights. Training therefore fits the former and moves away from fitting the latter.

Negative weights introduce their own danger. An ordinary squared-error loss penalizes prediction errors, but a negatively weighted term rewards increasing them. Without a restraint, the signed objective can decrease without bound and destabilize optimization.

SIFR constrains the average squared difference between the trainable model’s velocity predictions and those of a reference model. Its default reference is an exponential moving average of model parameters. The drift budget is calibrated from the pretrained model’s own flow-matching error and remains fixed during training.

An adaptive multiplier controls the strength of the drift penalty. When measured drift exceeds the budget, the multiplier increases; when drift falls below it, the multiplier decreases. This aims to permit useful learning without allowing uncontrolled changes. It controls average velocity drift, not every individual output or a direct measure of image quality.

The authors also derive a connection to existing methods, including DiffusionNFT and RAM: their update gradients can be expressed as signed regression with different reference regularizers. This is a mathematical equivalence of updates under the stated setups, not a claim that their original losses are identical.

Tests across three model sizes

The experiments use SD3.5 Medium with 2.5 billion parameters, FLUX.2 Klein with 9 billion, and Cosmos 3 Super Text2Image with 64 billion. All are fine-tuned using LoRA adapters rather than unrestricted updates to every model parameter.

Each backbone is tested on four objectives: PickScore alone; a combination of PickScore, HPSv2.1 and ClipScore; text rendering assessed through optical character recognition; and compositional generation assessed with GenEval. DiffusionNFT and RAM provide the main comparisons.

Preference-reward training generally improves faster with SIFR. In an extended Cosmos experiment, the baselines plateau near a held-out PickScore of 24.4. SIFR reaches that level in roughly one quarter as many training rounds and continues to approximately 25.36 by round 2,000. These are comparisons of training rounds, not demonstrated fourfold reductions in elapsed computing time.

For rule-based tasks, all three methods approach saturation. SIFR has the highest reported peak score in five of six backbone–task comparisons. The exception is SD3.5 Medium text rendering, where its 0.984 score is slightly below DiffusionNFT’s 0.985. On Cosmos GenEval, SIFR first reaches 0.97 at evaluation round 140, compared with 260 for DiffusionNFT and 290 for RAM.

Gains beyond the optimized score

After multi-reward training, SIFR achieves the highest ImageReward and HPSv3 scores on held-out prompts across all three backbones. On Cosmos, it leads all four held-out evaluators and all seven reported rewards on three additional prompt sets, including long captions.

Transfer is not universal. On smaller backbones, some alternative evaluators favor a baseline. Optimizing PickScore alone also does not consistently produce the best results on other metrics.

Ablations support the importance of both components. Removing negative advantages causes collapse even with the drift penalty in the tested setting. An adaptive multiplier also finishes above five tested fixed penalty coefficients on FLUX multi-reward training.

What the results establish—and do not

The evidence comes from a preprint and relies on automated rewards and benchmarks, rather than a reported direct human preference study. Some smaller-model comparisons use different samplers and evaluation guidance. RAM’s FLUX multi-reward tables use its round-300 checkpoint because that run later collapses.

The theoretical analysis holds rollout distributions fixed and derives conditions for stable constrained solutions; it does not establish unconditional stability for practical neural-network training. Several rule-based paired comparisons also have uncertainty intervals spanning an even split among non-tied cases.

The supported conclusion is narrower than universal superiority: adaptive control of signed updates improves reward optimization across the tested image generators, with substantial but evaluator-dependent transfer.