Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds

Researchers have introduced Intrinsic Muon (iMuon), a novel optimization framework designed to address limitations in existing norm-constrained matrix optimizers when applied to manifold-valued parameters. Traditional Muon optimizers, effective in Euclidean spaces, struggle with low-rank factorizations, orthogonality constraints, and symmetric positive definite matrices due to broken quotient symmetries and coupled constraints. The iMuon framework resolves these issues by leveraging Riemannian metrics to lift unitarily invariant Euclidean norms to intrinsic norms on tangent spaces, preserving symmetry and enabling closed-form updates. This approach is applicable to fixed-rank, SPD, Stiefel, and Grassmann manifolds using spectral, Frobenius, or nuclear norms. The study establishes convergence guarantees for both deterministic and stochastic variants, with rate constants dependent only on manifold dimension or rank, thereby eliminating the need for runtime factor-rescaling. Experimental results demonstrate the efficacy of iMuon in large-scale learning tasks, including LoRA finetuning of Large Language Models, image classification, and subspace learning, marking a significant advancement in geometric optimization for machine learning.