Mixture-of-Experts (MoE) has become a standard approach for scaling large language models, since each token only activates a small subset of experts and thereby reduces inference cost. However, the decode phase of MoE models remains highly memory-bound, meaning that memory access, rather than computation, is the limiting factor.
The arXiv paper "3D-MoE" introduces a hardware-software co-design strategy that targets this problem using 3D-stacked near-memory processing (NMP). The approach is described as "complementarity-guided," suggesting that the design exploits complementary strengths of hardware and software to improve decode efficiency.
This is a single-source digest based on the paper's abstract; no conflicting sources were available. The abstract does not provide quantitative results, but the framing points to a growing focus on memory-centric solutions for MoE inference as models continue to scale.