Guard系统可实时发现慢节点，提升大规模训练效率。
Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training
- 结合在线监控与离线检测，识别传统方法漏掉的缓慢故障
- 使平均浮点运算利用率提升1.7倍，训练步数波动从20%降至1%
- 适合超大规模模型训练团队，降低运维和调试成本
训练前沿规模的基础模型需协调数十万张GPU进行数月运行，即使轻微性能下降也会累积成显著效率损失。现有健康检查机制（如NCCL测试或GPU烧机）主要关注功能正确性，难以发现导致性能缓慢下降的‘慢故障’行为。本文提出Guard系统，通过训练期间轻量级在线性能监控，结合离线节点扫描机制，在节点参与生产任务前系统评估并筛选其健康状态。该设计可同时检测突发故障与长期慢故障，传统诊断无法捕捉。在大规模基础模型预训练中部署后，Guard将平均浮点运算利用率最高提升1.7倍，训练步数波动率从20%降至1%，延长平均故障间隔时间（MTTF），大幅降低运维与调试开销。结果表明，主动检测慢节点与系统化节点资格认证对保障大规模训练稳定高效至关重要。
原文摘要 · Abstract (English)
Training frontier-scale foundation models involves coordinating tens of thousands of GPUs over multi-month runs, where even minor performance degradations can accumulate into substantial efficiency losses. Existing health-check mechanisms, such as NCCL tests or GPU burn-in, primarily focus on functional correctness and often fail to detect fail-slow behaviors that silently degrade system performance. In this paper, we present Guard, a scalable system for detecting stragglers and ensuring node health in large-scale training clusters. Guard combines lightweight online performance monitoring during training with an offline node-sweep mechanism that systematically evaluates and qualifies nodes before they participate in production workloads. This design enables Guard to detect both acute failures and long-running fail-slow behaviors that traditional diagnostics cannot capture. Deployed on large-scale foundation model pretraining workloads, Guard improves mean FLOPs utilization by up to 1.7x, reduces run-to-run training step variance from 20% to 1%, increases mean time to failure (MTTF), and significantly reduces operational and debugging overhead. These results demonstrate that proactive straggler detection and systematic node qualification are critical for maintaining stable and efficient large-scale training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。