arXiv:2605.16311cs.LGcs.DC2026-05被引 2

1-bit通信优化器Sign-Muon,提升分布式训练效率。

SignMuon: Communication-Efficient Distributed Muon Optimization

  • 用符号+多数投票聚合梯度,仅传1比特信息
  • 330组实验中达92.15%准确率,提速37%
  • 适合大规模分布式训练,尤其资源受限场景

大型神经网络的分布式训练受全精度梯度通信和忽略权重张量矩阵结构的坐标优化器限制。我们提出Sign-Muon:一种1比特、矩阵感知的优化器,结合signSGD的多数投票符号聚合与Muon的极性步框架。每个工作节点通过牛顿-舒尔茨迭代获取动量的极性方向,仅传输逐元素符号,并通过多数投票聚合;可选的本地极性步骤在无额外通信成本下增强正交性。在谱范数光滑性和有界方差随机梯度假设下,谱范数归一化的符号步长对ℓ₁基平稳性测量实现𝒪(1/√T)的非凸收敛率。在单峰对称噪声下,M个节点的多数投票使随机项降低至1/√M,与signSGD一致。在α-β模型中,分布式Sign-Muon每轮仅需一次整数sum-allreduce,所有正交化操作均为本地计算,相比float32降低32倍带宽(int8为4倍)。在330组CIFAR-10/ResNet-50配置中,Sign-Muon达到最高验证准确率92.15%;其4-GPU多数投票变体在匹配有效批量下训练时间减少37%,准确率达92.02%。在nanoGPT上,Sign-Muon的困惑度更低,任意时刻性能优于其他基于符号的基线,且在16个GPU下表现良好弱扩展性。

原文摘要 · Abstract (English)

Distributed training of large neural networks is bottlenecked by full-precision gradient communication and by coordinatewise optimizers that ignore the matrix structure of weight tensors. We propose Sign-Muon, a 1-bit, matrix-aware optimizer that combines majority-vote sign aggregation from signSGD with the polar-step framework of Muon. Each worker forms a Muon-style direction by taking the polar factor of its momentum via a Newton--Schulz iteration, transmits only the entrywise signs, and aggregates by majority vote; an optional local polar step further enforces orthogonality at no extra communication cost. Under spectral-norm smoothness and bounded-variance stochastic gradients, the spectral-norm normalized sign step yields an $\mathcal{O}(1/\sqrt{T})$ nonconvex rate for an $\ell_1$-based stationarity measure. With unimodal symmetric noise, majority vote across $M$ workers cuts the stochastic term by $1/\sqrt{M}$, matching signSGD. In the $α$-$β$ model, distributed Sign-Muon needs only one integer sum-allreduce per iteration; all orthogonalization is local, giving a $32\times$ bandwidth reduction over float32 ($4\times$ for int8). Across 330 CIFAR-10/ResNet-50 configurations Sign-Muon attains the best validation accuracy (92.15\%); its 4-GPU majority-vote variant reaches 92.02\% with 37\% less training time at matched effective batch. On nanoGPT, Sign-Muon achieves lower perplexity and better anytime performance than other sign-based baselines, with favorable weak-scaling up to 16 GPUs.

分布式训练1比特优化通信压缩高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。