提出DiLoCoX框架，实现1070亿参数模型在1Gbps网络上高效去中心化训练。
DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster
- 结合流水线并行与双优化器策略，实现通信重叠与本地训练。
- 在1Gbps网络上训练1070亿参数模型，速度比传统AllReduce快357倍。
- 适合资源受限的分布式集群，首次实现超百亿参数去中心化训练。
大模型尤其是语言模型的分布式训练依赖高带宽通信，通常需集中式高速集群。能否在低速网络下利用去中心化集群训练超过1000亿参数的模型？本文提出DiLoCoX，一种低通信量的大规模去中心化训练框架。它融合流水线并行、双优化器策略、单步延迟通信重叠与本地训练、自适应梯度压缩机制。理论分析证明了通信重叠与梯度压缩的收敛性优势。实验表明，DiLoCoX可在1Gbps网络上预训练1070亿参数模型，相比原始AllReduce方法实现357倍加速，且模型收敛性能几乎无损失。据我们所知，这是首个成功应用于超百亿参数模型的去中心化训练框架。
原文摘要 · Abstract (English)
The distributed training of foundation models, particularly large language models (LLMs), demands a high level of communication. Consequently, it is highly dependent on a centralized cluster with fast and reliable interconnects. Can we conduct training on slow networks and thereby unleash the power of decentralized clusters when dealing with models exceeding 100 billion parameters? In this paper, we propose DiLoCoX, a low-communication large-scale decentralized cluster training framework. It combines Pipeline Parallelism with Dual Optimizer Policy, One-Step-Delay Overlap of Communication and Local Training, and an Adaptive Gradient Compression Scheme. This combination significantly improves the scale of parameters and the speed of model pre-training. We justify the benefits of one-step-delay overlap of communication and local training, as well as the adaptive gradient compression scheme, through a theoretical analysis of convergence. Empirically, we demonstrate that DiLoCoX is capable of pre-training a 107B foundation model over a 1Gbps network. Compared to vanilla AllReduce, DiLoCoX can achieve a 357x speedup in distributed training while maintaining negligible degradation in model convergence. To the best of our knowledge, this is the first decentralized training framework successfully applied to models with over 100 billion parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。