Distilling reasoning chains from large language models into smaller ones is a common way to create efficient models. But this usually requires full access to the teacher model, including its internal representations. The new arXiv paper, "Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing," addresses cases where researchers only have black-box access.
The proposed method combines two techniques: entropy gain sampling, which likely selects the most informative examples, and answer mixing, which combines multiple outputs to create richer training signals. The recursive aspect suggests the process is repeated, possibly using the distilled model as a new teacher.
The abstract does not provide experimental results or implementation details, so the effectiveness of the approach is not yet known. The paper is available on arXiv under identifier 2610.20914.