提出快速低成本的大型模型训练容错系统，150秒内恢复万级设备集群训练。
FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs
- 实时检测故障，秒级响应；
- 万设备集群150秒内恢复，与规模无关；
- 无需检查点，单步恢复，适合长时大规模训练。
大型语言模型（LLMs）因其强大能力在多个领域产生深远影响。然而，以空前规模训练这些模型需要庞大的AI加速器集群和复杂的并行策略，长期训练中系统可靠性面临严峻挑战。硬件和软件故障导致的训练时间损失尤为严重。为此，我们提出FlashRecovery，一种快速且低成本的故障恢复系统，包含三个核心模块：(1) 主动实时故障检测，通过持续监控训练状态，在数秒内识别软硬件故障，实现快速响应；(2) 与规模无关的任务重启，针对正常与故障节点采用不同恢复策略，并结合优化的通信组重构协议，确保恢复时间几乎不随集群规模变化；(3) 一步无检查点恢复，新型机制实现单步恢复，完全摆脱传统检查点方法及其开销。综合创新使FlashRecovery达到最优恢复时间目标（RTO）和恢复点目标（RPO），显著提升长周期大模型训练的可靠性与效率。实验表明，该系统可在4,800设备集群上于150秒内完成训练恢复，且不同规模任务的恢复时间基本一致。
原文摘要 · Abstract (English)
Large language models (LLMs) have made a profound impact across various fields due to their advanced capabilities. However, training these models at unprecedented scales requires extensive AI accelerator clusters and sophisticated parallelism strategies, which pose significant challenges in maintaining system reliability over prolonged training periods. A major concern is the substantial loss of training time caused by inevitable hardware and software failures. To address these challenges, we present FlashRecovery, a fast and low-cost failure recovery system comprising three core modules: (1) Active and real-time failure detection. This module performs continuous training state monitoring, enabling immediate identification of hardware and software failures within seconds, thus ensuring rapid incident response; (2) Scale-independent task restart. By employing different recovery strategies for normal and faulty nodes, combined with an optimized communication group reconstruction protocol, our approach ensures that the recovery time remains nearly constant, regardless of cluster scale; (3) Checkpoint-free recovery within one step. Our novel recovery mechanism enables single-step restoration, completely eliminating dependence on traditional checkpointing methods and their associated overhead. Collectively, these innovations enable FlashRecovery to achieve optimal Recovery Time Objective (RTO) and Recovery Point Objective (RPO), substantially improving the reliability and efficiency of long-duration LLM training. Experimental results demonstrate that FlashRecovery system can achieve training restoration on training cluster with 4, 800 devices in 150 seconds. We also verify that the time required for failure recovery is nearly consistent for different scales of training tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。