A new preprint on arXiv examines micro-expert pruning for mixture-of-experts (MoE) models. The method reduces model capacity by removing coupled channels within routed experts, a finer-grained alternative to coarser pruning approaches.
The abstract cautions that finer pruning granularity alone does not guarantee preservation of downstream performance. This suggests that the way channels are coupled and removed matters beyond simply choosing a smaller unit of pruning.
The paper is titled "Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models," indicating a hierarchical approach to micro-expert pruning as a potential solution to this limitation. Further details of the method and results are not available in the abstract.