Distilling reasoning from large teacher models into compact ones is a common way to build efficient medical assistants. But the rationales those teachers provide are not always correct, and a smaller model may faithfully learn the error rather than the answer. A new preprint examines this trade-off and proposes a controlled workflow that audits rationales before distillation.
The authors frame the problem as deciding when teacher rationales are worth transferring. Using Brazilian Portuguese health professions questions, they show that rationales can boost performance in some cases but can also entrench mistakes. Their workflow treats the rationale as evidence to be checked, not as ground truth, and uses that audit to guide whether and how the student model is trained.
The paper does not claim that auditing solves all distillation failures, but it offers a practical checkpoint for teams building compact medical language models. The focus on Portuguese-language data is a useful reminder that distillation methods need validation across languages and exam formats, not just on English benchmarks.