Distilling diffusion models into fewer sampling steps typically struggles when moving beyond one step: multistep distillation often requires training data, suffers from early-step degradation, and produces inconsistent outputs across different sampling budgets. The authors trace these problems to simultaneous training, where a single adapted network tries to approximate multiple score functions of different student distributions at once, leading to biased distribution matching and unstable optimization.

Their proposed method, Progressive Multistep Data-free Distillation (PMDD), assigns a dedicated adapted network to each student distribution while avoiding the memory overhead of maintaining multiple networks. A stage-wise unforget loss preserves earlier-step generations, and the result is a single unified model that supports multiple inference budgets with faster convergence and reduced optimization bias.

Experiments show PMDD reaches FID 1.99 on ImageNet 64x64 in two steps, below its 512-NFE teacher's 2.32, and achieves FID 7.76 on zero-shot COCO 512x512. It also requires 4-5x less training compute than data-free DMDv2 when supporting three inference budgets, while matching or beating data-free few-step baselines at every multistep budget.