解决分布式训练中通信稀疏导致的优化性能下降问题
MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
- 设计多时尺度动量机制，同步慢速与快速更新信号
- 在语言模型预训练中实现与全同步训练相当的困惑度表现
- 适合跨数据中心、广域地理分布的大规模模型训练
使用分布式数据并行训练大模型需要频繁在工作节点间通信梯度，易饱和带宽。稀疏通信策略（如Local SGD）虽降低开销，但应用于自适应优化器时，性能常落后于完全同步的分布式数据并行（DDP）。我们发现此差距源于时间尺度不匹配：优化器快速变化的动量项为高频更新设计，无法在长间隔内有效平滑梯度，导致噪声主导优化过程。为此，我们提出MT-DAO，一类采用多个慢速与快速移动的一阶矩或梯度来追踪不同时间尺度更新动态的优化器，并首次提供收敛性保证。实验表明，在语言模型预训练中，该方法消除了与DDP的性能差距，在困惑度上优于稀疏通信基线，并在以太网互联下将等令牌墙钟时间减少6%-27%。在720M规模下，MT-DAO比单动量DDP基线提前24%步数达到目标困惑度，耗时减少35%。该方法支持有效的跨数据中心训练及广域地理分布下的模型训练。
原文摘要 · Abstract (English)
Training large models with distributed data parallelism (DDP) requires frequent communication of gradients across workers, which can saturate bandwidth. Infrequent communication strategies (e.g., Local SGD) reduce this overhead but, when applied to adaptive optimizers, often suffer a performance gap relative to fully synchronous DDP. We trace this gap to a time-scale mismatch: the optimizer's fast-moving momentum, tuned for frequent updates, decays too quickly to smooth gradients over long intervals, leading to noise-dominated optimization. To address this, we propose MT-DAO, a family of optimizers that employs multiple slow- and fast-moving first momenta or the gradient to track update dynamics across different time scales, for which we provide the first convergence guarantees. Empirically, for language-model pre-training, this eliminates the performance gap with DDP, outperforming infrequent-communication baselines in perplexity and reducing iso-token wall-clock time by 6-27% on Ethernet interconnects. At the 720M scale, MT-DAO reaches a target perplexity in 24% fewer steps and 35% less time than the single-momentum DDP baseline. MT-DAO enables effective cross-datacenter training and training over wide geographic areas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。