arXiv:2506.02007cs.DCcs.AI2025-06被引 3

无需修改代码,实时监控AI系统软硬件性能并检测异常

eACGM: Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems

  • 基于eBPF无侵入式采集GPU、网络及CUDA等关键组件数据
  • 用高斯混合模型识别延迟异常、通信效率低下等复杂故障模式
  • 适用于大规模分布式训练场景,支持生产环境故障诊断

我们提出eACGM,一种基于eBPF的全栈AI/ML系统监控框架。eACGM无需代码注入或修改,即可实时采集GPU、网络通信层以及CUDA、Python、PyTorch等软件栈的关键性能数据;同时利用libnvml获取进程级GPU资源使用情况。通过将多维性能指标输入高斯混合模型(GMM)进行统计建模与聚类分析,eACGM可有效识别延迟异常、硬件故障和通信低效等复杂失效模式,实现系统瓶颈与异常行为的快速定位。我们在多节点分布式训练场景中进行了大量实证研究与案例分析,结果表明:eACGM在保持非侵入性与低开销的同时,成功捕捉了模型训练与推理过程中的关键性能异常。其稳定的异常检测表现与全面的监控能力,验证了其在真实生产环境中用于性能优化与故障诊断的适用性与可扩展性。

原文摘要 · Abstract (English)

We present eACGM, a full-stack AI/ML system monitoring framework based on eBPF. eACGM collects real-time performance data from key hardware components, including the GPU and network communication layer, as well as from key software stacks such as CUDA, Python, and PyTorch, all without requiring any code instrumentation or modifications. Additionally, it leverages libnvml to gather process-level GPU resource usage information. By applying a Gaussian Mixture Model (GMM) to the collected multidimensional performance metrics for statistical modeling and clustering analysis, eACGM effectively identifies complex failure modes, such as latency anomalies, hardware failures, and communication inefficiencies, enabling rapid diagnosis of system bottlenecks and abnormal behaviors. To evaluate eACGM's effectiveness and practicality, we conducted extensive empirical studies and case analyses in multi-node distributed training scenarios. The results demonstrate that eACGM, while maintaining a non-intrusive and low-overhead profile, successfully captures critical performance anomalies during model training and inference. Its stable anomaly detection performance and comprehensive monitoring capabilities validate its applicability and scalability in real-world production environments, providing strong support for performance optimization and fault diagnosis in large-scale AI/ML systems.

系统监控异常检测eBPFAI运维

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。