T-MUON: Tensorized Optimizer for LLM Pre-training

A new optimizer extends Muon's matrix-based approach to tensors via t-SVD, aiming to improve large language model pre-training.

3 November 2026 · 1 min read · 1 source

Muon has recently emerged as a strong optimizer for large language model pre-training, leveraging matrix structure through polar orthogonalization. However, it processes each parameter matrix independently, potentially missing correlations between matrices. The new paper T-MUON proposes a tensorized variant that applies t-SVD to address this limitation.

The provided abstract is truncated, so full methodological details and experimental outcomes are not available in the source. The contribution is a conceptual extension of Muon to tensor form, with the stated aim of improving pre-training. The paper is announced on arXiv under identifier 2610.24938.

Source

01T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-trainingarXiv

T-MUON: Tensorized Optimizer for LLM Pre-training

Source

More in Research Digest

Polynomial-Time Learning Shown for Positive ReLU Networks Under Gaussian Inputs

Sequence-Level Policy Optimization with Conformal Prediction for Reasoning LLMs

ClinSDT adds LLM semantic guidance to Decision Transformers for sepsis

Optimal Sample Size Found for Pointwise Replicable Learning