Knowledge distillation (KD) is a common way to improve language-model pretraining by matching a teacher model's output distributions. However, the abstract for a new arXiv paper notes that standard KD only provides supervision at corpus prefixes — the fixed contexts from the training data. It offers no direct guidance at contexts that the student model itself generates during training.

The proposed method, PA1-OPD, addresses this gap by adding one-step prefix-augmented on-policy distillation. Rather than relying solely on teacher distributions at static prefixes, it also supervises the student at contexts produced by its own predictions. This on-policy signal is designed to provide more relevant feedback during pretraining.

The paper is announced on arXiv (2610.19601) and describes the approach as a way to strengthen KD for language model pretraining. As the abstract is brief, the reported details are limited to the method's motivation and design; no experimental results are included in the available text.