揭示梯度正交化背后的优化机制，统一多种高效训练算法。
Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- 将梯度正交化视为非欧信任域优化，用矩阵谱范数定义更新范围。
- 提出新算法，在多种函数类型下实现领先收敛速度。
- 解释了Muon优于Orthogonal-SGDM及权重衰减对大模型的重要性。
近期研究表明，矩阵梯度正交化在深度神经网络训练中表现优异（Jordan等，2024；Liu等，2025）。本文提供理论分析，表明正交化梯度方法可视为一种一阶信任域优化，其信任域基于矩阵谱范数定义。受此启发，我们提出了带动量的随机非欧信任域梯度方法，该方法涵盖Muon优化器（Jordan等，2024）、归一化SGD与带动量的signSGD（Cutkosky and Mehta, 2020；Sun等，2023）作为特例。此外，我们在多种场景下证明了所提算法的最优收敛性，包括任意非欧范数、约束与复合问题，以及非凸、星凸、一阶与二阶光滑函数。最后，理论结果解释了若干实践现象，如Muon相比Tuddenham等（2022）提出的Orthogonal-SGDM具有实际优势，以及权重衰减在大规模语言模型训练中的关键作用。
原文摘要 · Abstract (English)
Optimization with matrix gradient orthogonalization has recently demonstrated impressive results in the training of deep neural networks (Jordan et al., 2024; Liu et al., 2025). In this paper, we provide a theoretical analysis of this approach. In particular, we show that the orthogonalized gradient method can be seen as a first-order trust-region optimization method, where the trust-region is defined in terms of the matrix spectral norm. Motivated by this observation, we develop the stochastic non-Euclidean trust-region gradient method with momentum, which recovers the Muon optimizer (Jordan et al., 2024) as a special case, along with normalized SGD and signSGD with momentum (Cutkosky and Mehta, 2020; Sun et al., 2023). In addition, we prove state-of-the-art convergence results for the proposed algorithm in a range of scenarios, which involve arbitrary non-Euclidean norms, constrained and composite problems, and non-convex, star-convex, first- and second-order smooth functions. Finally, our theoretical findings provide an explanation for several practical observations, including the practical superiority of Muon compared to the Orthogonal-SGDM algorithm of Tuddenham et al. (2022) and the importance of weight decay in the training of large-scale language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。