提出急切更新机制，让通信与计算重叠，提升分布式训练效率。
Eager Updates For Overlapped Communication and Computation in DiLoCo
- 采用急切更新策略，使外层同步与内层计算并行进行。
- 在低带宽环境下性能接近标准DiLoCo，通信延迟显著降低。
- 适合大规模分布式训练场景，尤其适用于数据中心间通信受限的场景。
分布式优化方法如DiLoCo已被证明在跨多个分布式工作节点（如数据中心）训练超大规模模型时非常有效。这些方法将更新过程分为两部分：内层优化阶段，各工作节点在其本地数据上独立执行多步优化；外层优化步骤，同步内层更新结果。尽管此类方法相比标准数据并行训练所需通信量减少数个数量级，但在工作节点为数据中心的场景下，即使通信开销较低，每次外层优化步骤仍需阻塞等待，造成显著延迟。本文研究通过将通信与计算重叠的方式缓解此问题，实现外层优化步骤与内层优化阶段完全重叠。我们发现一种特定变体——急切更新（eager updates），在低带宽条件下可达到与标准DiLoCo相当的性能。
原文摘要 · Abstract (English)
Distributed optimization methods such as DiLoCo have been shown to be effective in training very large models across multiple distributed workers, such as datacenters. These methods split updates into two parts: an inner optimization phase, where the workers independently execute multiple optimization steps on their own local data, and an outer optimization step, where the inner updates are synchronized. While such approaches require orders of magnitude less communication than standard data-parallel training, in settings where the workers are datacenters, even the limited communication requirements of these approaches can still cause significant slow downs due to the blocking necessary at each outer optimization step. In this paper, we investigate techniques to mitigate this issue by overlapping communication with computation in a manner that allows the outer optimization step to fully overlap with the inner optimization phase. We show that a particular variant, dubbed eager updates, provides competitive performance with standard DiLoCo in settings with low bandwidth between workers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。