Scaling model parameters increasingly relies on sparsely activated Mixture-of-Experts (MoE), which activates only a subset of parameters per input. Most mainstream MoE models are hierarchical, meaning each layer has its own set of experts. That design can lead to large storage footprints and slow retrieval.
The new preprint from arXiv proposes a method called Meta Expert, which shares experts across layers rather than keeping them separate. The goal is to reduce the storage and retrieval overhead typically associated with hierarchical MoE while preserving the benefits of sparse activation.
Because the source abstract is truncated, further experimental details and quantitative results are not available in this excerpt. The paper is described as a new announcement, so the approach and its reported implications should be read from the full preprint.