On-policy distillation (OPD) is a standard technique for transferring behavior from a large teacher model to a smaller student. According to a new preprint, however, OPD carries an implicit assumption: that the teacher's and student's predictions can be compared token by token. This assumption holds when both models share a tokenizer, but it breaks down in cross-tokenizer settings.
The paper, titled "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation," argues that this mismatch causes supervision to be lost during training. The authors propose SimCT as a way to recover that lost supervision, making distillation feasible even when teacher and student tokenizers are not aligned.
The abstract does not provide implementation details, so the exact mechanism behind SimCT is not described in this source. This is a single preprint, and no independent commentary or comparison with other methods is included here. The significance lies in identifying a previously overlooked assumption in a widely used technique and offering a targeted fix.