提出更鲁棒的非欧优化器MuonMax，结合动量提升性能
An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
- 将多种梯度范数聚合方式统一为非欧梯度下降框架
- 新变体MuonMax对学习率不敏感，表现更稳定
- 结合模型动量（Momo）后显著降低调参成本
为定义神经网络的最速下降法，需选择每层的范数、跨层范数的聚合方式及是否归一化。本文系统探索了跨层范数聚合的不同方案，形式化了Adam与近期提出的Muon作为非欧梯度下降的特例，并推导出Muon的新变体。在全面实验评估中发现，Muon对学习率敏感，而新提出的MuonMax则显著更鲁棒。进一步展示如何将任意非欧梯度方法与基于模型的动量（Momo）结合。新推出的Momo-Muon变体对超参数调优更具鲁棒性，且常取得更优验证分数。因此，在新任务中，若最优超参数未知，建议使用结合MuonMax的Momo以节省昂贵的调参成本。
原文摘要 · Abstract (English)
To define a steepest descent method over a neural network, we need to choose a norm for each layer, a way to aggregate these norms across layers, and whether to use normalization. We systematically explore different alternatives for aggregating norms across layers, both formalizing existing combinations of Adam and the recently proposed Muon as a type of non-Euclidean gradient descent, and deriving new variants of the Muon optimizer. Through a comprehensive experimental evaluation of the optimizers within our framework, we find that Muon is sensitive to the choice of learning rate, whereas a new variant we call MuonMax is significantly more robust. We then show how to combine any non-Euclidean gradient method with model based momentum (known as Momo). The new Momo variants of Muon are significantly more robust to hyperparameter tuning, and often achieve a better validation score. Thus for new tasks, where the optimal hyperparameters are not known, we advocate for using Momo in combination with MuonMax to save on costly hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。