Many deployed distillation pipelines show the student a truncated view of the teacher: keep the top-k probabilities, renormalize, and minimize a divergence on what remains. The assumption is that renormalization accounts for the missing probability mass, making the truncated distribution a valid teaching signal.
A new arXiv preprint (2610.15589) challenges that assumption. The paper argues that this procedure never computes a particular gradient term—the one that would arise from the missing mass itself. By ignoring that term, the student may be optimizing a subtly different objective than intended.
The abstract is brief, so the full derivation and experimental evidence are not yet available. Still, the claim points to a potential blind spot in a widely used technique, and it invites practitioners to re-examine whether renormalization truly preserves the teacher's intent.