A new paper on arXiv presents oracle inequalities for dense and sparse mixture-of-experts (MoE) models. MoE systems combine specialized experts through input-dependent routing, and sparse variants are widely used in large-scale machine learning. The analysis focuses on whether the adaptive routing that enables sparsity also carries a statistical penalty.
The paper's central finding, as stated in its title, is that sparsity saves compute but not samples. In other words, the number of training examples required to reach a given accuracy does not improve with sparse routing compared to a dense model. The authors provide theoretical bounds to quantify this trade-off, though the abstract is truncated and does not give full details of the methods.
The result is significant because sparse MoE models are often assumed to be more efficient overall. This work suggests that the efficiency gain is purely computational, not statistical—a distinction that matters for understanding when such models are worth the added complexity.