A new preprint on arXiv challenges a common practice in training large-scale diffusion models. According to the abstract, these models are usually trained with a constant learning rate and an exponential moving average (EMA). This approach persists even though learning-rate decay and Schedule-Free optimization have shown strong performance when training large language models.
The authors point to this contrast as a potential gap. While the abstract does not detail the proposed alternative, it implies that diffusion models could benefit from adopting techniques that have proven effective in the LLM domain. The paper's title — "Large Diffusion Models Need More Than EMA" — signals that the standard recipe may be suboptimal.
If the findings hold, they could lead to more efficient or higher-quality diffusion training. But the abstract alone does not provide experimental results, so the practical impact remains to be seen. The preprint is available at arXiv:2610.22945.