Shampoo优化器比Muon更高效，关键在于对权重矩阵的特殊处理。
Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory
- 将Shampoo更新分解为修正版Muon，揭示其核心机制。
- 在语言模型上，Shampoo比Muon多节省30%训练数据量。
- 适合关注优化器设计与参数结构关系的研究者。
利用神经网络中矩阵结构的优化器（如Shampoo和Muon）比元素级算法（如Adam和Signum）更具数据效率。尽管在特定情况下，Shampoo和Muon可退化为谱下降，类似于Adam和Signum退化为符号下降，但二者在受控设置下的相对数据效率仍不明确。我们在语言模型上进行大量实验，发现Shampoo的词元效率高于Muon，与Adam优于Signum的现象一致。我们证明，对权重矩阵应用的Shampoo更新可分解为一种改进的Muon更新。与此一致，Shampoo的优势完全归因于其对权重矩阵的应用，挑战了忽略参数形状的解释。这提出了一种新视角：与谱下降强制半正交性不同，Shampoo的更新在期望下是时间平均半正交的。
原文摘要 · Abstract (English)
Optimizers leveraging the matrix structure in neural networks, such as Shampoo and Muon, are more data-efficient than element-wise algorithms like Adam and Signum. While in specific settings, Shampoo and Muon reduce to spectral descent analogous to how Adam and Signum reduce to sign descent, their general relationship and relative data efficiency under controlled settings remain unclear. Through extensive experiments on language models, we demonstrate that Shampoo achieves higher token efficiency than Muon, mirroring Adam's advantage over Signum. We show that Shampoo's update applied to weight matrices can be decomposed into an adapted Muon update. Consistent with this, Shampoo's benefits can be exclusively attributed to its application to weight matrices, challenging interpretations agnostic to parameter shapes. This admits a new perspective that also avoids shortcomings of related interpretations based on variance adaptation and whitening: rather than enforcing semi-orthogonality as in spectral descent, Shampoo's updates are time-averaged semi-orthogonal in expectation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。