为大模型专家并行通信设计统一接口，支持高吞吐与低延迟场景。
NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
- 基于NCCL设备端API构建，原生支持跨节点通信。
- 低延迟模式下小批量（1-128令牌）性能媲美专用库。
- 适合需要高效专家并行的训练与推理场景，如vLLM集成。
混合专家（MoE）架构已成为扩展大语言模型的关键，推动了如DeepEP、Hybrid-EP等由GPU发起的专用通信库的发展。本文提出NCCL EP（专家并行），一个完全基于NCCL设备API从头构建的MoE通信库。它提供统一的ncclEpDispatch和ncclEpCombine接口，支持C与Python调用，包含低延迟（LL）模式用于推理解码，以及高吞吐（HT）模式用于训练和推理预填充。LL模式针对小批量（1-128个令牌）设计，利用直接all-to-all RDMA+NVLink网格连接，并通过双缓冲重叠调度与合并阶段。HT模式针对大批量（4096+令牌）设计，采用分层通信，在NVLink域内聚合令牌后再进行跨节点RDMA传输。两种模式均使用设备端API实现节点内与跨节点通信，发挥其拓扑感知与优化的GPU发起机制优势。我们在基于H100的多节点集群上评估了NCCL EP，展示了具有竞争力的低延迟内核性能，并提供了与vLLM集成的端到端结果。通过在当前及未来NVIDIA平台上原生构建专家并行通信，NCCL EP提供了一条受支持的专家并行路径。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。