提出实用化去中心化训练方法，显著提升多机训练效率
From promise to practice: realizing high-performance decentralized training
- 设计可提速的去中心化通信架构与计算模式
- 64卡集群实测显示迭代耗时更短、泛化更好
- 适合大规模Transformer模型高效训练场景
去中心化深度神经网络训练因理论上的可扩展性优势而备受关注，优于同步数据并行方法（如All-Reduce）。然而在多节点训练中实现其潜力面临挑战，涉及通信拓扑、计算模式与优化算法的复杂设计空间。本文识别出三个能带来加速的关键因素，并构建运行时模型，以确定去中心化在何时、如何及在何种程度上能实现更短的每轮迭代时间。为支持基于Transformer模型的去中心化训练，研究了一种可重叠通信与计算的去中心化Adam算法，证明其收敛性，并提出一种累积技术以缓解小本地批量带来的高方差问题。在最多64张GPU的集群上部署该方法，验证了其在固定迭代预算下的实际可行性和性能优势。
原文摘要 · Abstract (English)
Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability over synchronous data-parallel methods like All-Reduce. However, realizing this potential in multi-node training is challenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs a runtime model to determine when, how, and to what degree decentralization can yield shorter per-iteration runtimes. Furthermore, to support the decentralized training of transformer-based models, we study a decentralized Adam algorithm that allows for overlapping communications and computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy the proposed approach in clusters with up to 64 GPUs and demonstrate its practicality and advantages in both runtime and generalization performance under a fixed iteration budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。