Knowledge distillation for vision transformers often involves transferring attention maps from a pretrained teacher to a student model. A recent result claimed that simply transferring the teacher's attention patterns to a randomly initialized student is enough to recover the teacher's full performance, suggesting a cheap and effective distillation shortcut.

The new paper, titled "Attention Transfer Is Not Universally Effective for Vision Transformers," pushes back on that conclusion. The authors argue that the earlier claim is too broad, and that attention transfer does not work in all settings. While the abstract is brief, the title itself signals that the method's success depends on conditions that the original work may have overlooked.

This matters for researchers and engineers who rely on attention transfer as a lightweight alternative to full distillation. If the approach only works under specific circumstances, its use in practice requires more careful validation rather than assuming it will always succeed.