分析大模型推理中集体通信模式，揭示网络拥堵根源
Collective Communication Profiling of Modern-day Machine Learning Workloads
- 通过增强NVIDIA通信库日志，追踪多种模型的通信行为
- 发现通信操作类型、数据量分布存在显著差异，峰值带宽超预期
- 适合网络优化与分布式训练系统设计者参考
大规模机器学习任务在分布式高性能系统上运行时，频繁使用AllReduce、AllGather、Broadcast等集体通信操作，易引发高带宽和突发流量，导致网络拥塞和丢包，影响整体性能。为此，本文对多种模型（如DeepSeek、GPT、Llama等）的集体通信行为进行广泛分析。通过扩展NVIDIA Collective Communication Library的日志功能，获取更丰富的上下文信息，并调整并行度、节点数、模型类型等配置参数。本研究以开源DeepSeek V3推理模型为例，展示了通信操作类型与数量、各操作传输数据大小及请求规模分布等关键指标。结果表明，现有通信框架与网络拓扑需重新评估，以应对网络异常对上述工作负载的影响。
原文摘要 · Abstract (English)
Machine Learning jobs, carried out on large number of distributed high performance systems, involve periodic communication using operations like AllReduce, AllGather, and Broadcast. These operations may create high bandwidth and bursty traffic patterns, leading to network congestion and packet loss, thus impacting the performance of these jobs. Hence it is imperative to analyze these patterns, which can be helpful in provisioning network resources depending on the type of machine learning workloads. In this poster we carry out extensive analysis of the collective communication behavior seen in a wide variety of models (ex. DeepSeek, GPT, Llama, etc.) To achieve this we instrument Nvidia Collective Communication Library logging functionality for richer context about the collectives and workloads. We adjust configuration parameters that influence collective communication behavior, such as parallelism, number of nodes, and model type. This overview presents and discusses some of the results on the collective communication behavior for the open source DeepSeek V3 inferencing model, which includes operation type and count, transfer sizes per operation, and request size distribution. Our analysis shows that it makes sense to rethink current collective communication frameworks and network topologies so as to accommodate the effect of network anomalies on the mentioned workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。