On-policy distillation is a common way to merge multiple specialist models into a single student, but it is expensive: each teacher demands rollouts, forward passes, and gradient updates. A new arXiv preprint argues that this consolidation step does not actually need training at all.

The authors report that the same consolidation can be achieved with forward passes only, removing the need for gradient steps and the associated backpropagation. If the claim holds, it would cut a major computational bottleneck out of model merging pipelines.

The abstract does not yet spell out the exact mechanism or experimental results, so the practical limits of the approach remain unclear. Still, the idea of training-free distillation is a striking departure from the standard paradigm and worth watching as full details emerge.