DeMuon首次实现图上去中心化矩阵优化的可证明复杂度，适合分布式训练场景。
DeMuon: A Decentralized Muon for Matrix Optimization over Graphs
- 基于牛顿-舒尔兹正交化与梯度追踪，解决局部函数异质性问题
- 在重尾噪声下达到最优迭代复杂度，逼近容忍度依赖最优
- 适用于分布式Transformer预训练，对不同连通性网络均表现更优
本文提出DeMuon，一种在给定通信拓扑上进行去中心化矩阵优化的方法。DeMuon通过牛顿-舒尔兹迭代实现矩阵正交化——这一技术源自其集中式前身Muon——并采用梯度追踪以缓解局部函数间的异质性。在重尾噪声条件下，并在额外温和假设下，我们建立了DeMuon达到近似随机平稳点的迭代复杂度。该复杂度结果在目标容差依赖关系上与现有最优集中式算法一致。据我们所知，DeMuon是首个在图上实现去中心化优化且具有可证明复杂度保证的Muon直接扩展。我们在具有不同连通性的图上进行了去中心化Transformer预训练的初步数值实验。结果表明，DeMuon在不同网络拓扑下相较其他主流去中心化算法均有明显性能提升。
原文摘要 · Abstract (English)
In this paper, we propose DeMuon, a method for decentralized matrix optimization over a given communication topology. DeMuon incorporates matrix orthogonalization via Newton-Schulz iterations-a technique inherited from its centralized predecessor, Muon-and employs gradient tracking to mitigate heterogeneity among local functions. Under heavy-tailed noise conditions and additional mild assumptions, we establish the iteration complexity of DeMuon for reaching an approximate stochastic stationary point. This complexity result matches the best-known complexity bounds of centralized algorithms in terms of dependence on the target tolerance. To the best of our knowledge, DeMuon is the first direct extension of Muon to decentralized optimization over graphs with provable complexity guarantees. We conduct preliminary numerical experiments on decentralized transformer pretraining over graphs with varying degrees of connectivity. Our numerical results demonstrate a clear margin of improvement of DeMuon over other popular decentralized algorithms across different network topologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。