通过模拟实验分析大模型训练中的慢节点问题，揭示其成因与影响。
Understanding Stragglers in Large Model Training Using What-if Analysis
- 用反事实分析对比有无慢节点的训练表现
- 发现慢节点每月影响超10%的任务，延迟达数小时
- 适合分布式系统和大模型训练优化的研究者
大规模语言模型（LLM）训练是当今最复杂的分布式计算之一，通常需要数千个GPU并频繁跨机器同步。这种工作模式极易受慢节点影响，导致训练停滞。在字节跳动的实践中，我们发现慢节点并非仅由硬件故障引起，而是由多种复杂因素共同导致。本研究基于字节跳动LLM训练集群五个月的运行日志，开展全面分析。核心方法为反事实分析：模拟不存在慢节点的情况，并与实际运行对比。我们重点回答三个问题：(1) 慢节点对训练任务的影响频率及性能损失；(2) 慢节点是否存在时间或空间上的聚集模式；(3) 潜在的根本原因是什么。分析表明，慢节点每月影响超过10%的任务，平均延迟可达数小时，且存在明显的时空聚集性。
原文摘要 · Abstract (English)
Large language model (LLM) training is one of the most demanding distributed computations today, often requiring thousands of GPUs with frequent synchronization across machines. Such a workload pattern makes it susceptible to stragglers, where the training can be stalled by few slow workers. At ByteDance we find stragglers are not trivially always caused by hardware failures, but can arise from multiple complex factors. This work aims to present a comprehensive study on the straggler issues in LLM training, using a five-month trace collected from our ByteDance LLM training cluster. The core methodology is what-if analysis that simulates the scenario without any stragglers and contrasts with the actual case. We use this method to study the following questions: (1) how often do stragglers affect training jobs, and what effect do they have on job performance; (2) do stragglers exhibit temporal or spatial patterns; and (3) what are the potential root causes for stragglers?
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。