AdaMuon优化器通过自适应正交更新，显著提升大规模训练效率。
AdaMuon: Adaptive Muon Optimizer
- 结合元素级自适应与正交更新，保持稳定梯度方向。
- 在大规模训练中比Adam快40%以上，无需调整学习率。
- 适合追求高训练效率的深度学习研究者与工程师。
我们提出AdaMuon，一种新型优化器，将逐元素自适应与正交更新相结合，用于大规模神经网络训练。AdaMuon包含两个紧密耦合机制：(1) 在正交化更新方向上应用逐元素二阶动量估计器；(2) 采用符号稳定型正交更新，先对动量进行符号变换再进行正交化。这两个组件共同实现方差自适应缩放，同时保持稳定的更新几何结构。此外，AdaMuon采用与RMS对齐的重新缩放策略，使均方根更新幅度与Adam一致，从而可直接复用现有学习率调度，无需额外调参。实验表明，AdaMuon不仅保持训练稳定性，且在大规模场景下训练效率超越Adam超过40%。
原文摘要 · Abstract (English)
We propose AdaMuon, a novel optimizer that combines element-wise adaptivity with orthogonal updates for large-scale neural network training. AdaMuon incorporates two tightly coupled mechanisms: (1) an element-wise second momentum estimator applied to orthogonalized update directions, and (2) a sign-stabilized orthogonal update, where the momentum is first sign-transformed before orthogonalization. These two components jointly enable variance-adaptive scaling while maintaining stable update geometry. In addition, AdaMuon employs an RMS-aligned rescaling strategy to match the root-mean-square update magnitude to Adam, allowing direct reuse of existing learning rate schedules without extra tuning. Experiments demonstrate that AdaMuon not only maintains stability but can surpass Adam by more than 40\% training efficiency in large-scale scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。