Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard way to improve Mixture-of-Experts (MoE) large language models, but it has a known limitation: the rewards are only given at the outcome level, which is sparse and can leave many tokens without direct learning signal. A new arXiv paper (2610.23185) tackles this by introducing On-Policy Distillation, a method that generates dense token-level supervision to complement the sparse outcome rewards.

The core idea is to use a stronger teacher within the model itself, rather than relying on an external, larger model. The paper describes this as "bidirectional distillation," where the teacher and student are derived from the same MoE model, and the distillation happens on-policy—meaning the supervision is aligned with the model's current generation distribution. This allows the training signal to reach every token, not just the final outcome.

The authors argue that this approach strengthens MoE models more effectively than RLVR alone, because it fills the gap between outcome-level rewards and the need for fine-grained token-level guidance. Since this is a single source, there are no conflicting findings to compare, but the proposal points to a promising direction for making MoE training more sample-efficient and robust. Further details on the exact architecture and empirical results are not included in the abstract, so the practical gains remain to be verified in the full paper.