改进正交动量优化器的稳定性，提升训练效率与鲁棒性。
TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers
- 通过全局RMS校准和能量信任区域裁剪，保留正交几何同时稳定更新幅度。
- 在视觉与语言模型上收敛速度优于基线，无需预热阶段也表现稳定。
- 适合对优化器鲁棒性要求高的场景，如大模型训练或敏感超参调优。
Muon类优化器利用Newton-Schulz（NS）迭代实现更新正交化，其更新几何常优于Adam系列方法。然而，该正交化过程丢弃了幅值信息，导致训练对学习率超参数敏感且易受高能突变干扰。为缓解此问题，本文提出TrasMuon（Trust Region Adaptive Scaling Muon）。TrasMuon在保持Muon近似等距几何的同时，通过（i）全局均方根校准和（ii）基于能量的信任区域裁剪来稳定更新幅度。实验表明，尽管自适应缩放可提升优化效率，但通常会因高能异常值加剧不稳定性。TrasMuon通过相对能量比定义信任区域，将更新限制在稳定区间内。在视觉与语言模型上的实验证明，TrasMuon收敛更快；无预热阶段的实验进一步验证其卓越的稳定性和鲁棒性。
原文摘要 · Abstract (English)
Muon-style optimizers leverage Newton-Schulz (NS) iterations to orthogonalize updates, yielding update geometries that often outperform Adam-series methods. However, this orthogonalization discards magnitude information, rendering training sensitive to step-size hyperparameters and vulnerable to high-energy bursts. To mitigate this, we introduce TrasMuon (\textbf{T}rust \textbf{R}egion \textbf{A}daptive \textbf{S}caling \textbf{Muon}). TrasMuon preserves the near-isometric geometry of Muon while stabilizing magnitudes through (i) global RMS calibration and (ii) energy-based trust-region clipping. We demonstrate that while reintroducing adaptive scaling improves optimization efficiency, it typically exacerbates instability due to high-energy outliers. TrasMuon addresses this by defining a trust region based on relative energy ratios, confining updates to a stable zone. Empirical experiments on vision and language models demonstrate that TrasMuon converges faster than baselines. Furthermore, experiments without warmup stages confirm TrasMuon's superior stability and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。