跨数据中心训练大模型，提速17倍并提升94%显卡利用率。
Improving training time and GPU utilization in geo-distributed language model training
- 通过动态带宽分配与任务调度优化，减少训练延迟。
- 在显卡空闲时段运行预填充服务，利用率最高达94%。
- 适合大规模分布式训练场景，尤其对资源受限团队有价值。
语言模型（LM）的广泛应用导致对GPU需求激增。训练大型语言模型需数万张GPU，集中于同一数据中心面临峰值功耗等限制。本文研究通过广域网（WAN）连接多个数据中心进行训练。提出Atlas系统，采用新型工作负载感知的时序带宽共享机制，显著缩短训练时间。尽管如此，仍存在显卡空闲周期（bubbles）。为此设计BubbleTea，在空闲时段运行预填充即服务（prefill-as-a-service，LM推理的一部分），有效提升显卡利用率，且不影响训练过程。相比现有方案，Atlas与BubbleTea联合使用可实现最高17倍加速，显卡利用率高达94%。代码将开源。
原文摘要 · Abstract (English)
The widespread adoption of language models (LMs) has caused a huge surge in demand for GPUs. Training large LMs requires tens of thousands of GPUs and housing them in the same datacenter (DC) is a challenge due to many constraints including availability of peak power. We focus on training such models across multiple DCs connected via the Wide-Area-Network (WAN). We built Atlas that speeds up the training time using novel workload-aware temporal bandwidth sharing and other design choices. While Atlas improves the training time, it does not completely eliminate the bubbles (idle GPU cycles). We built BubbleTea that runs prefill-as-a-service (part of LM inference) during the bubbles thus improving the GPU utilization without any impact on training. Compared to state-of-the-art designs, Atlas and BubbleTea together achieve up to 17x faster training, and up to 94% GPU utilization. The code will be open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。