用实时探针+智能分析，6分钟定位大规模训练中的通信故障GPU
CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training
- 通过轻量级分布式追踪监测通信流量，实现跨层异常指标采集
- 在4000卡集群上部署，99%以上异常被覆盖，定位时间<6分钟
- 适合大模型训练团队排查通信慢/卡死问题，提升调试效率
随着训练规模扩大，集体通信库（CCL）面临由硬件、软件和环境因素复杂交互引发的异常。这些异常通常表现为通信慢或卡死，是最常见且最难诊断的类型。传统方法精度低、效率差，常需数小时甚至数天完成根因分析。为此，我们提出CCL-D，一个高精度诊断系统，用于检测和定位大规模分布式训练中的通信慢/卡死异常。CCL-D结合了逐节点实时探针与智能决策分析器：探针利用轻量级分布式追踪框架测量跨层异常指标，监控通信流量；分析器自动完成异常检测与根因定位，精准识别出故障GPU节点。在4000个GPU的集群上持续运行一年，CCL-D实现了对已知慢/卡死异常的近乎全覆盖，并在6分钟内精确定位故障节点，显著优于现有方案。
原文摘要 · Abstract (English)
As training scales grow, collective communication libraries (CCL) increasingly face anomalies arising from complex interactions among hardware, software, and environmental factors. These anomalies typically manifest as slow/hang communication, the most frequent and time-consuming category to diagnose. However, traditional diagnostic methods remain inaccurate and inefficient, frequently requiring hours or even days for root cause analysis. To address this, we propose CCL-D, a high-precision diagnostic system designed to detect and locate slow/hang anomalies in large-scale distributed training. CCL-D integrates a rank-level real-time probe with an intelligent decision analyzer. The probe measures cross-layer anomaly metrics using a lightweight distributed tracing framework to monitor communication traffic. The analyzer performs automated anomaly detection and root-cause location, precisely identifying the faulty GPU rank. Deployed on a 4,000-GPU cluster over one year, CCL-D achieved near-complete coverage of known slow/hang anomalies and pinpointed affected ranks within 6 minutes-substantially outperforming existing solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。