A new preprint on arXiv introduces "Bridged Self-Distillation for On-Policy Learning," a method for transferring reasoning capabilities from a strong teacher model to a smaller student model. The work focuses on on-policy distillation, a setup where the student learns from the teacher's behavior during training.
The abstract notes that distillation from a stronger teacher faces two mismatches: the teacher's solutions may be too difficult for a smaller student to replicate. The authors propose a bridged self-distillation approach to address these issues, though the available abstract does not detail the exact mechanism.
As an arXiv announcement, the paper has not yet been peer-reviewed, and the abstract provides only a high-level summary. Readers interested in the technical specifics will need to consult the full manuscript.