优化多节点大模型推理通信，显著降低延迟
Understanding and Improving Communication Performance in Multi-node LLM Inference
- 提出分层归约算法NVRAR，基于NVSHMEM改进通信效率
- 在128KB-2MB消息下，延迟比NCCL低1.9至3.6倍
- 集成后使405B模型端到端延迟降低1.72倍，适合大规模推理部署
随着大语言模型规模持续增长，分布式推理变得愈发重要。模型并行策略需高效扩展至多GPU乃至多节点环境。本文针对基于GPU超算的多节点分布式推理开展性能研究，对比多个前沿推理引擎与自研原型引擎YALIS。分析不同模型并行方案的强可扩展性，识别关键瓶颈。由于全部归约操作普遍存在性能瓶颈，我们设计了基于递归加倍与NVSHMEM的分层全部归约算法NVRAR。在HPE Slingshot和InfiniBand互连环境下，对于128KB至2MB的消息大小，NVRAR延迟比NCCL降低1.9×–3.6×。集成至YALIS后，在使用张量并行的多节点解码密集型负载中，对Llama 3.1 405B模型实现最高1.72×的端到端批处理延迟下降。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to grow in size, distributed inference has become increasingly important. Model-parallel strategies must now efficiently scale not only across multiple GPUs but also across multiple nodes. In this work, we present a detailed performance study of multi-node distributed inference using LLMs on GPU-based supercomputers. We conduct experiments with several state-of-the-art inference engines alongside YALIS, a research-oriented prototype engine designed for controlled experimentation. We analyze the strong-scaling behavior of different model-parallel schemes and identify key bottlenecks. Because all-reduce operations are a common performance bottleneck, we develop NVRAR, a hierarchical all-reduce algorithm based on recursive doubling with NVSHMEM. NVRAR achieves up to 1.9$\times$-3.6$\times$ lower latency than NCCL for message sizes between 128 KB and 2 MB on HPE Slingshot and InfiniBand interconnects. Integrated into YALIS, NVRAR achieves up to a 1.72$\times$ reduction in end-to-end batch latency for the Llama 3.1 405B model in multi-node decode-heavy workloads using tensor parallelism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。