On-policy self-distillation (OPSD) allows large language models to learn from privileged information (PI) without requiring a separate teacher model. A new preprint on arXiv examines a gap in this approach: PI is usually selected as a fixed design choice, and its actual impact on learning is not well understood.

The study centers on predicting the effectiveness of privileged information, which would make OPSD less reliant on arbitrary choices. According to the abstract, prior work has largely left the effects of PI unexamined, so this appears to be an early step toward principled PI selection.

Because the announcement does not include full details, the specific prediction method and experimental results are not available from the source. The main contribution evident so far is the framing of PI effectiveness as a problem worth predicting, rather than treating PI as a constant.