Muon has gained attention as a powerful optimizer for neural network training, with a theoretical interpretation as steepest descent under a spectral-norm geometry. However, the authors of a new arXiv paper report that Muon-trained Transformers can degrade sharply when their MLP components are trained under this scheme, pointing to a practical instability in a common architecture.
To address this, the paper introduces BMuon, a method that budgets Muon's spectral geometry for structured compressibility. Rather than discarding Muon's geometric foundation, BMuon modifies how the spectral norm is applied, aiming to preserve the optimizer's strengths while preventing the sharp performance drops seen in MLP layers.
The abstract does not provide full experimental details, but the framing suggests that the fix is targeted at the interaction between Muon's update rule and the structure of Transformer MLPs. The result is a more stable training trajectory without abandoning the spectral-norm perspective that makes Muon attractive in the first place.