面向超大规模训练的实时故障诊断系统，可精准定位软硬件问题
EROICA: Online Performance Troubleshooting for Large-scale Model Training
- 通过在线采样与差异可观测性，实现全集群细粒度性能监控
- 在百万级GPU集群上运行1.5年，97.5%故障诊断成功率
- 适合运维团队、训练平台研发者快速排查生产环境性能瓶颈
大规模模型训练（LMT）的性能故障排查极为困难，源于现代GPU集群的空前规模、软硬件交互复杂性以及训练过程的数据密集性。现有针对传统分布式系统或数据中心网络的排查方法难以适用。本文提出EROICA，首个可在生产环境中运行的在线故障诊断系统，支持基于采样的细粒度观测与全集群覆盖，可诊断包括软硬件问题及其混合情况在内的各类性能异常。EROICA通过在线采样总结LMT函数执行的运行行为模式，并利用差异可观测性实现根因定位，对生产环境影响极小。该系统已在约10万张GPU的大规模集群上作为生产服务部署1.5年，成功诊断多种复杂性能问题，整体诊断准确率达97.5%。
原文摘要 · Abstract (English)
Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditional distributed systems or datacenter networks fall short and can hardly apply to real-world training systems. In this paper, we present EROICA, the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems (or the mixture of both). EROICA effectively summarizes runtime behavior patterns of LMT function executions via online profiling, and leverages differential observability to localize the root cause with minimal production impact. EROICA has been deployed as a production service for large-scale GPU clusters of ~100,000 GPUs for 1.5 years. It has diagnosed a variety of difficult performance issues with 97.5% success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。