通过分离动量信号，实现低通信量的大模型分布式训练。
Distributed Low-Communication Training with Decoupled Momentum Optimization
- 将Nesterov动量分解为高低频分量，仅同步高频部分。
- 相比基线方法通信量降低16倍，支持多种模型架构。
- 适合在低带宽网络环境下训练大模型的研究者使用。
大型模型的训练需要大量计算资源，通常仅限于具备高带宽互联的数据中心。然而，减少节点间对高带宽互联的依赖，可使分布式计算资源成为数据中心训练的替代方案。基于分布式模型训练的最新进展，我们提出一种进一步降低通信开销的方法：结合稀疏同步与梯度动量压缩。具体而言，将优化器动量视为信号，通过离散余弦变换（DCT）将其分解为高频和低频分量，仅每H步同步高频分量。实验表明，该方法相比基线DiLoCo实现最高16倍的通信减少，并在包括Transformer语言模型和图像卷积神经网络在内的多种架构上具有泛化能力。本工作显著提升了在低带宽互联环境下训练大规模模型的可行性。
原文摘要 · Abstract (English)
The training of large models demands substantial computational resources, typically available only in data centers with high-bandwidth interconnects. However, reducing the reliance on high-bandwidth interconnects between nodes enables the use of distributed compute resources as an alternative to centralized data center training. Building on recent advances in distributed model training, we propose an approach that further reduces communication by combining infrequent synchronizations across distributed model replicas with gradient momentum compression. In particular, we treat the optimizer momentum as a signal and decompose the Nesterov momentum into high- and low-frequency components via the discrete cosine transform (DCT). Only the high-frequency components are synchronized across model replicas every $H$ steps. Empirically, our method achieves up to a $16\times$ reduction in communication compared to the baseline DiLoCo, and it generalizes across architectures, including transformer-based language models and convolutional neural networks for images. Overall, this work advances the feasibility of training large models on distributed nodes with low-bandwidth interconnects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。