Dion让大模型训练更高效，通过低秩正交更新减少计算开销。
Dion: Distributed Orthonormalized Updates
- 用动量缓冲区的幂迭代替代矩阵重构，适配分片权重架构
- 在160M到3B参数模型上提速显著，降低大规模训练耗时
- 支持低秩更新与误差反馈，适合下一代大模型优化
正交化更新能加速训练、提升稳定性并实现超参数鲁棒迁移，但现有方法如Muon依赖密集矩阵运算，在大规模语言模型分片权重训练中导致高计算与通信开销。我们提出Dion（分布式正交化），将牛顿-舒尔兹迭代替换为动量缓冲区上的摊销幂迭代，避免全矩阵重建，并与权重分片无缝集成。通过引入秩分数参数与误差反馈，实现低秩更新，在保证性能的同时大幅降低成本。在160M至3B参数的语言模型上，Dion保持正交化更新优势，且在规模化下显著减少实际运行时间，是下一代基础模型的实用优化器。代码见：https://github.com/microsoft/dion/
原文摘要 · Abstract (English)
Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash with sharded weights in large-scale LLM training, causing high compute and communication cost. We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule that replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer, avoiding full-matrix reconstruction and integrating cleanly with weight sharding. The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings. On language models from 160M to 3B parameters, Dion retains the benefits of orthonormalized updates, while markedly reducing wall-clock time at scale, making it a practical optimizer for next-generation foundation models. Code is available at: https://github.com/microsoft/dion/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。