用DPU实时监测大模型推理中的负载不均问题
A Study of Skews, Imbalances, and Pathological Conditions in LLM Inference Deployment on GPU Clusters detectable from DPU
- 将监控任务卸载到DPU，分析GPU状态和节点通信
- 发现多卡推理中存在显著的计算与通信不平衡
- 适合关注大模型部署性能优化的研究者
基于Transformer的大规模语言模型（LLM）自回归推理在运行时效率上面临挑战，尤其在解码阶段，各GPU分片间的负载不均会导致吞吐量下降和延迟飙升。本文利用BlueField-3数据处理单元（DPU）构建辅助框架，实现多节点张量并行推理中负载不均的实时检测与缓解。通过将监控任务卸载至DPU，分析GPU遥测数据及节点间通信模式，系统可向推理控制器和调度器提供可操作反馈。本研究旨在三方面：(i) 识别多GPU执行中出现的偏差、不平衡及异常状况（涵盖训练与推理阶段的LLM张量计算），(ii) 分析其对计算性能的影响，(iii) 评估这些现象是否可通过DPU网络进行追踪以实现潜在缓解。
原文摘要 · Abstract (English)
Autoregressive inference in large transformer-based language models (LLMs) presents significant challenges for runtime efficiency, particularly during the decode phase where load imbalance across GPU shards can cause throughput degradation and latency spikes. A DPU-assisted framework leveraged by BlueField-3 Data Processing Units can enable real-time detection and mitigation of load imbalance in multi-node tensor-parallel inference. By offloading monitoring tasks to the DPU and analyzing GPU telemetry and inter-node communication patterns, the resulting system can provide actionable feedback to inference controllers and schedulers. The goal of this study is three-fold i) identify the reported skews/imbalances/pathological conditions that arise in muti-GPU execution of a) LLM tensor computing (both during training and inference), b) identify their impact on computational performance, and c) make a critical assessment if those can be tracked for potential mitigation from a DPU network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。