On-Policy Distillation: Teacher Gating vs. Privileged Info
Two papers probe how much a frozen teacher's knowledge and reliability matter when a student model learns from its own rollouts.
Both arXiv papers address the same family of methods: on-policy distillation, in which a student language model trains on its own rollouts while a frozen teacher provides dense token-level supervision. The two share a starting concern—that vanilla on-policy distillation leaves accuracy on the table—but they take different cuts at the problem.
The first paper, What Does Privileged Information Add to On-Policy Self-Distillation?, asks whether the teacher's effectiveness improves when it has access to an answer or a worked solution. The abstract notes that such extra information seems to give the student more to learn, and the paper appears to test how much of that promise is real. It focuses on the teacher's input.
The second paper, Verify Before You Distill, focuses instead on the teacher's output. It observes that vanilla on-policy distillation applies supervision uniformly across all prompts, without checking whether the teacher is reliable for each one. The authors propose prompt-level gating: verify the teacher on a prompt before allowing it to supervise the student.
So the two papers agree that naive on-policy distillation is suboptimal, but they diverge on where the fix should go. One enriches the teacher's information; the other selectively silences the teacher when it might mislead. Read together, they frame a design trade-off for distillation pipelines: whether to improve what the teacher knows or when it gets to speak.
Sources · 7
- Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation
- CLOOPD: Closing the Learner Loop in On-Policy Distillation
- Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
- Data-free On-policy Distillation
- Calibrating Teacher--Student Discrepancy for On-Policy Distillation
- What Does Privileged Information Add to On-Policy Self-Distillation?
- Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
More in Research Digest
OpenRoIS middleware decouples HRI apps from robot hardware
A new open-source middleware aims to let human-robot interaction applications run across different robot platforms without rewriting code.
4DGS-Fixer: Video Diffusion Priors for Sparse-View 4D Scenes
A new method iteratively refines 4D Gaussian splatting with video diffusion priors to improve dynamic scene synthesis from sparse-view videos.
Two papers probe RL bottlenecks for LLM agents
New arXiv work targets two stubborn obstacles in reinforcement learning for LLMs: sparse credit in open-ended tasks and merging separately trained capabilities.
LLM Social Agents Need More Than Mimicry, Two Preprints Argue
Two new arXiv papers identify a common reliability gap in LLM-driven social simulations, then propose different fixes: tuned personalities versus a Bayesian belief layer.