字节跳动打造高稳定训练系统，应对万卡规模大模型训练故障
Robust LLM Training Infrastructure at ByteDance
- 基于大模型训练特性设计容错机制，主动检测并恢复故障
- 万卡集群三月训练任务实现97%设备可用率，故障响应快速
- 适合大规模训练平台运维人员与分布式系统研发者参考
大语言模型（LLM）训练规模已扩展至数万张GPU，持续增长带来大量故障（如CUDA错误、数值溢出、任务挂起等），严重威胁训练稳定性。为保障大规模训练连续高效，基础设施需具备极低中断率、快速故障诊断和强容错能力。本文提出面向大模型训练的鲁棒性管理框架ByteRobust，针对训练过程特点，优先处理故障检测与恢复。利用大模型训练中的并行特性，实现高容量容错、快速故障定位与归因，结合数据驱动方法全面保障训练连续性。该系统部署于生产级GPU平台，在9,600张GPU上完成为期三个月的训练任务，达到97%的设备可用率（ETTR）。
原文摘要 · Abstract (English)
The training scale of large language models (LLMs) has reached tens of thousands of GPUs and is still continuously expanding, enabling faster learning of larger models. Accompanying the expansion of the resource scale is the prevalence of failures (CUDA error, NaN values, job hang, etc.), which poses significant challenges to training stability. Any large-scale LLM training infrastructure should strive for minimal training interruption, efficient fault diagnosis, and effective failure tolerance to enable highly efficient continuous training. This paper presents ByteRobust, a large-scale GPU infrastructure management system tailored for robust and stable training of LLMs. It exploits the uniqueness of LLM training process and gives top priorities to detecting and recovering failures in a routine manner. Leveraging parallelisms and characteristics of LLM training, ByteRobust enables high-capacity fault tolerance, prompt fault demarcation, and localization with an effective data-driven approach, comprehensively ensuring continuous and efficient training of LLM tasks. ByteRobust is deployed on a production GPU platform and achieves 97% ETTR for a three-month training job on 9,600 GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。