用硬件计数器实现跨框架、跨精度的GPU效率实时监控。
Instant GPU Efficiency Visibility at Fleet Scale
- 基于芯片级计数器构建无侵入式效率指标，兼容不同显卡和精度。
- 在608个生产任务中与实际效率相关性达0.78，误差小于2个百分点。
- 适合运维团队持续监测大规模集群的推理与训练效率问题。
我们提出总体浮点运算利用率（OFU），一种面向高性能计算系统上AI工作负载的硬件级、精度无关的GPU效率度量方法，基于两个片上性能计数器：张量管道活动和流多处理器时钟频率。OFU无需应用层埋点，适用于不同代际GPU和数值精度。通过在H100和GB200上对FP16、TF32、FP8、NVFP4进行受控的GEMM实验，分析了五种影响因素：块量化、浮点精度缩放、时钟采样噪声、张量核心时钟域及非张量操作漏计。经块量化校正后，OFU预测应用级MFU误差不超过2个百分点。在608个生产训练任务中，OFU与应用级MFU的相关系数达到r=0.78，揭示了两个框架级的浮点运算计算错误。在大规模GPU集群部署中，成功检测到2.5倍的效率下降，并追踪到混合精度预训练中的精度依赖型利用率变化。评估与运行经验表明，OFU是应用级MFU在全舰队持续监控中的实用补充。
原文摘要 · Abstract (English)
We present Overall FLOP Utilization (OFU), a hardware-level, precision-agnostic GPU efficiency metric for AI workloads on HPC systems, derived from two on-chip performance counters: Tensor Pipe Activity and SM clock frequency. OFU requires no application instrumentation and works across GPU generations and numeric precisions. We characterize five properties of the OFU approximation -- tile quantization, floating-point precision scaling, clock sampling noise, Tensor Core clock domains, and non-tensor undercounting -- through controlled GEMM experiments on H100 and GB200 across FP16, TF32, FP8, and NVFP4. After tile-quantization correction, OFU predicts application-level MFU to within <=2 percentage points. Against 608 production training jobs, OFU achieves r = 0.78 correlation with application-level MFU and surfaces two framework-level FLOPs miscalculations. Deployed across large-scale GPU fleets, OFU has detected a 2.5x efficiency regression and tracked precision-dependent utilization changes in mixed-precision pretraining. Our evaluation and operational experience suggest OFU is a practical, deployment-ready complement to application-level MFU for continuous fleet-wide efficiency monitoring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。