Sparse routing in mixture-of-experts models cuts per-token computation, but the full set of experts still has to stay resident in memory. That memory burden is especially relevant for speech-centric omni-modal models, which must handle multiple modalities without blowing up deployment costs.
A new arXiv paper, PW-Pruning, proposes a phase-aware expert selection strategy to prune experts. The authors argue that pooling calibration statistics can obscure differences between experts, so they instead account for the phase of processing when deciding which experts to keep.
The work directly targets the memory bottleneck left by sparse routing. By pruning experts, the model can retain the computational benefits of sparse routing while reducing the memory required to store the expert bank. The abstract is truncated, so specific experimental results are not yet available in the source.