A new paper on arXiv, “Ask to Distill: Learning to Synthesize Data for On-Policy Distillation,” examines how few training questions are needed for on-policy distillation (OPD). The authors report that OPD can succeed with surprisingly few examples, which leads them to suggest that questions themselves are not just prompts but carriers of distributional knowledge transfer.
The abstract then turns to a central methodological question: how can we assess whether a newly generated question is valuable? The paper appears to address this by learning to synthesize data, though the abstract leaves the specific evaluation criteria unspecified in the available text.
This framing shifts attention from the quantity of training data to the quality and informational content of individual questions. For researchers working on distillation, the work points toward a more principled way of deciding which synthetic questions are worth using.