Mixture-of-experts (MoE) models make a sparse choice for every token: which experts should process it. A new arXiv preprint, NR-SAE, asks whether a sparse dictionary should reuse that same choice. The idea is to give each routed expert a small dictionary, rather than learning a single global dictionary.
The abstract states that NR-SAE gives each routed expert a small dictionary, but the description is cut off, so the full method and results are not available from the source. The significance is conceptual: it connects the routing mechanism in MoE models to sparse dictionary learning, potentially reducing the cost of sparse coding.
Because the abstract is truncated, we cannot report on performance or comparisons. The source only establishes the motivation and the design choice of per-expert dictionaries. Further details would require the full paper.