DiLoCo让大模型训练更省通信，还能越练越好。
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- 用降低同步频率的方法实现高效分布式训练
- 相同算力下，模型越大效果越优，小模型也胜过传统方法
- 适合大规模模型训练者，尤其关注通信效率的团队
随着机器学习模型规模不断增大，数据并行训练中频繁的同步操作带来显著延迟，成为进一步扩展的瓶颈。近期提出的DiLoCo方法通过放松同步要求，在不损失模型质量的前提下缓解了这一问题。然而，现有研究未深入分析其在不同模型规模下的表现变化。本文在固定计算预算下研究了DiLoCo训练大语言模型的缩放规律，重点考察模型副本数、超参数及分词预算等算法因素如何影响训练效果，并发现这些影响可通过缩放定律准确预测。结果表明，DiLoCo在模型规模上表现出可预测且稳健的缩放特性：调优后，其性能随模型增大优于传统数据并行训练，甚至在小模型时也能超越。此外，相较于传统方法，DiLoCo能支持更大有效批量、提升模型泛化能力，并在固定分词预算下获得更低的评估损失。
原文摘要 · Abstract (English)
As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work develops an approach (DiLoCo) that relaxes synchronization demands without compromising model quality. However, these works do not carefully analyze how DiLoCo's behavior changes with model size. In this work, we study the scaling law behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on how algorithmic factors, including number of model replicas, hyperparameters, and token budget affect training in ways that can be accurately predicted via scaling laws. We find that DiLoCo scales both predictably and robustly with model size. When well-tuned, DiLoCo scales better than data-parallel training with model size, and can outperform data-parallel training even at small model sizes. Our results showcase a more general set of benefits of DiLoCo than previously documented, including increased optimal batch sizes, improved downstream generalization with scale, and improved evaluation loss for a fixed token budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。