On-policy self-distillation has become a popular way to train reasoning models, using a privileged self-teacher to provide dense, step-by-step supervision. However, as a new arXiv preprint argues, this supervision is limited: it reflects local token preferences rather than the expected return of an unguided student. That means the teacher may reinforce choices that look good at each step but do not lead to better overall outcomes.
The paper introduces VASPO (Self-Distillation with Privileged Value Estimation), a method designed to address this gap. By incorporating value estimation into the distillation process, VASPO aims to shift the supervision signal from purely local preferences toward a more global measure of expected return. The abstract suggests this correction helps the student learn more effectively, though full experimental details are not yet available in the provided text.
This is a preprint, so the claims have not yet been peer-reviewed. Still, the problem it tackles—misalignment between dense local feedback and long-term reward—is a known challenge in training reasoning models, and the proposed solution appears to be a direct response to that issue.