10万张GPU训练时，故障容忍新方案让恢复时间从10分钟降到3分钟。
Training LLMs with Fault Tolerant HSDP on 100,000 GPUs
- 用数据副本做容错，单副本出错只停该副本，其余继续训练。
- 故障恢复耗时从10分钟降至3分钟，有效训练时间提升至80%。
- 适合超大规模训练，尤其在高故障率的千卡级集群中使用。
大规模训练系统通常采用同步训练，要求所有GPU同时健康运行。在我们训练百万级GPU规模的经验中，同步训练因频繁故障和长恢复时间导致效率低下。为此，我们提出一种新型训练范式——故障容忍混合共享数据并行（FT-HSDP）。FT-HSDP以数据并行副本为容错单元，当发生故障时，仅将包含故障GPU或服务器的单个副本下线重启，其余副本持续训练。为实现这一理念，FT-HSDP引入多项技术：1）提出故障容忍全归约协议（FTAR），由CPU处理动态参与者增删等复杂控制逻辑，由GPU负责高效数据传输；2）设计非阻塞追赶协议，使恢复中的副本能快速接入训练，最小化停顿。与十万级GPU下的全同步训练相比，FT-HSDP将故障恢复导致的停机时间从10分钟降至3分钟，有效训练时间从44%提升至80%。进一步实验表明，异步恢复不会造成模型精度下降。
原文摘要 · Abstract (English)
Large-scale training systems typically use synchronous training, requiring all GPUs to be healthy simultaneously. In our experience training on O(100K) GPUs, synchronous training results in a low efficiency due to frequent failures and long recovery time. To address this problem, we propose a novel training paradigm, Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP). FT-HSDP uses data parallel replicas as units of fault tolerance. When failures occur, only a single data-parallel replica containing the failed GPU or server is taken offline and restarted, while the other replicas continue training. To realize this idea at scale, FT-HSDP incorporates several techniques: 1) We introduce a Fault Tolerant All Reduce (FTAR) protocol for gradient exchange across data parallel replicas. FTAR relies on the CPU to drive the complex control logic for tasks like adding or removing participants dynamically, and relies on GPU to perform data transfer for best performance. 2) We introduce a non-blocking catch-up protocol, allowing a recovering replica to join training with minimal stall. Compared with fully synchronous training at O(100K) GPUs, FT-HSDP can reduce the stall time due to failure recovery from 10 minutes to 3 minutes, increasing effective training time from 44\% to 80\%. We further demonstrate that FT-HSDP's asynchronous recovery does not bring any meaning degradation to the accuracy of the result model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。