A preprint posted on arXiv takes up the question of how experts are shaped in fine-grained mixture-of-experts (MoE) models. The authors note that common approaches increase the number of experts by narrowing each one, effectively treating expert shape as an incidental result of routing granularity rather than something worth designing directly.
The paper proposes LatentMoE, which moves expert computation into a lower-dimensional space. It also discusses expert geometry and layerwise capacity allocation, suggesting that these aspects can be shaped deliberately to improve how capacity is distributed across layers.
This is a single preprint, so the claims have not yet been peer-reviewed or independently verified. The abstract is truncated in the source, so details beyond the proposal and its motivation are limited.