Feature distillation is typically expected to improve performance when combined with logit distillation. In compact multilingual automatic speech recognition (ASR), however, the authors find that this expectation does not hold for two common feature losses.
To understand why, they analyze the interaction between feature and logit distillation in this setting. They then introduce CAFE (Confining Feature Distillation to Language-Routed Adapters), which restricts feature distillation to language-routed adapters rather than applying it across the whole model.
The work highlights that feature distillation is not universally beneficial in compact multilingual ASR, and that targeted application may be necessary to realize gains.