504 GPU集群训练中，故障检测与恢复效率成关键瓶颈。
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
- 通过多信号监测识别各类硬件故障，避免单一指标失效
- 检查点存取达700 GB/s读带宽的21.5%，写带宽使用率达16.0%
- 顶尖3个节点占故障排除超50%，自动重试成功率是人工的2.7倍
大规模AI训练本质上是分布式系统问题，硬件故障是常态而非例外，但生产集群的实际运行数据仍十分有限。本报告基于一个包含63节点（504块GPU）的NVIDIA B200生产集群，分析了55天的Prometheus时序数据和73天的操作日志，涵盖224次多节点训练任务。该环境由五方（SKT、Upstage、Lablup、NVIDIA Korea、VAST Data）共享统一监控管道，实现联合诊断。成功识别出在2-4节点测试中未暴露的60节点级存储I/O瓶颈，此现象仅靠单个团队无法定位。我们完成三项定量分析，得出四个发现：第一，超过751个Prometheus指标和10次XID标识的GPU故障中，无单一指标对所有故障类型均主导，支持多信号检测；第二，523次检查点事件追踪了从GPU显存到NFS服务器的存取路径，重启加载达到最大读带宽700 GB/s的21.5%，保存突发达最大写带宽250 GB/s的16.0%，且NFS/RPC队列与传输层积压同步上升；第三，在73天内224次会话中，节点剔除高度集中，前3个节点占比超50%；第四，自动重试链分析显示12条链共73次尝试中成功率为33.3%，是人工操作12.5%的2.7倍，中位重试间隔为11分钟（四分位距10–11）。所有分析依托生产级基础设施，具备会话级工作负载管理、以GPU为中心的调度与统一可观测性。
原文摘要 · Abstract (English)
Large-scale AI training is fundamentally a distributed systems problem, where hardware failures are routine operating conditions rather than rare exceptions, yet public operational evidence from production training clusters remains limited. This report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504 GPUs), using 55 days of Prometheus time-series data and 73 days of operational logs covering 224 multi-node training sessions. The environment is cross-organizational: five parties (SKT, Upstage, Lablup, NVIDIA Korea, VAST Data) share a unified monitoring pipeline. This enabled joint diagnosis of a 60-node-scale storage I/O bottleneck absent in 2-4-node tests, a production-scale phenomenon no single team could isolate alone. We perform three quantitative analyses yielding four findings. First, over 751 Prometheus metrics and 10 XID-identified GPU failures, no single metric is consistently dominant across failure types, motivating multi-signal detection. Second, 523 checkpoint events trace the save/load path from GPU VRAM to the NFS server: restart loading reaches 21.5% of maximum read bandwidth (700 GB/s) and save bursts 16.0% of maximum write bandwidth (250 GB/s), with NFS/RPC queueing and transport-layer backlog rising together. Third, across 224 sessions over 73 days, node exclusions concentrate so the top 3 of 63 nodes account for over 50%. Fourth, auto-retry chain analysis shows a 33.3% success rate over 12 chains (73 attempts), 2.7x the 12.5% manual rate, with a median retry interval of 11 minutes (IQR 10-11). All analyses are grounded in production infrastructure providing session-level workload management, GPU-centric scheduling, and unified observability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。