改进了归一化优化器的自适应步长,提升对超参数敏感度的鲁棒性。
Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization
- 基于轨迹探索距离自适应调整信任域半径,提升稳定性
- 在非凸与星凸目标下分别实现最优解收敛与渐近误差界
- 无需预知初始点到最优解距离,适合深度学习模型训练
Muon及其相关归一化优化器将更新方向与步长选择解耦,但实际性能仍对归一化步长尺度敏感。本文研究了一般范数几何下的自适应缩放规则,提出三种互补算法。针对光滑非凸目标,提出距离自适应的Muon,其信任域半径由轨迹探索范围设定,并在轨迹有界假设下证明了驻点保证。针对星凸目标(常用于解释深度神经网络损失景观),提出尺度校准的Muon,保留指数移动平均,但通过当前梯度与动量计算局部下降证来设定步长,在初始子集有界假设下证明了最后迭代点的O(1/T)目标间隙界,其中半径参数仅出现在分析中而不影响算法。进一步提出无距离的Muon,一种重新中心的信任域方法,使用标量距离证和一维上界搜索选择信任域半径,无需已知初始点至全局最小值的距离。在Transformer语言建模(GPT-124M/WikiText-103)和图像分类(ViT-Tiny/CIFAR-100)上的实验表明,所提自适应缩放规则降低了对人工尺度调优的依赖,在测试预算下表现匹配或优于调优后的固定尺度基准。
原文摘要 · Abstract (English)
Muon and related normalized optimizers decouple the choice of update direction from the choice of step scale, but their practical performance remains sensitive to the scale of the normalized step. We study adaptive scaling rules for Muon in general norm geometries and develop three complementary algorithms. For smooth non-convex objectives, we introduce Distance-Adaptive Muon, whose trust-region radius is set from the radius explored by the trajectory, and prove a stationarity guarantee under a bounded-trajectory assumption. We then turn to star-convex objectives, a tractable model of the favorable global geometry often used to reason about the empirical loss landscapes of deep neural networks, where objective-gap guarantees are possible. In this setting, we first introduce Scale-Calibrated Muon, which keeps Muon's exponential moving average but sets the step length from a local descent certificate computed from the current gradient and momentum. For this method, we prove a last-iterate O(1/T) objective-gap bound under a bounded initial sublevel-set assumption, where the corresponding radius parameter appears only in the analysis and not in the algorithm. Finally, we develop Distance-Free Muon, a recentered trust-region method that uses a scalar distance certificate and a majorized one-dimensional search to select the trust-region radius without requiring the unknown distance from the initialization to a global minimizer. Experiments on Transformer language modeling (GPT-124M/WikiText-103) and image classification (ViT-Tiny/CIFAR-100) show that the proposed adaptive scaling rules reduce sensitivity to manual scale tuning and match or improve tuned fixed-scale Muon baselines under the tested budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。