Reward fine-tuning is a common way to adapt pretrained diffusion and flow-matching models to specific downstream goals. But as the authors of a new arXiv preprint note, pushing optimization too hard can backfire: the model may collapse to a narrow set of outputs or drift away from its original distribution.
The paper, titled 'Distribution-Based Regularization for Reward Fine-Tuning Diffusion and Flow Models,' proposes a regularization strategy to keep this in check. The abstract explains that existing methods attempt to address the problem, though the description cuts off before detailing how they compare. The key idea, as the title suggests, is to use the distribution of the pretrained model as a reference point during fine-tuning.
Because the abstract is truncated, the specific mechanism and experimental results are not available in the source. The contribution, at least as stated, is a regularization approach aimed at preserving the model's original behavior while still optimizing the reward.