A long-standing disparity in deep learning is the focus of a new arXiv paper: while modern architectures naturally exhibit rich symmetry and equivariance, popular optimizers like Adam do not account for these geometric properties. The authors argue that this mismatch is striking and propose a principle to make optimizer design symmetry-compatible.

The principle is aimed at specific components of neural networks, including embeddings, language-model heads, SwiGLU MLPs, and mixture-of-experts routers. By aligning the optimizer with the symmetries of these structures, the approach may offer a more principled alternative to generic optimizers.

The abstract does not detail the resulting optimizer or provide experimental validation, so the practical impact of the principle remains to be seen. The paper is available as arXiv:2605.18106v5.