重构专家并行通信库，显著降低大模型推理延迟
UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
- 设计新型通信架构，打破传统同步瓶颈
- 实测推理延迟降低11.1%，全集合通信延迟降52.4%
- 适合大规模部署稀疏专家模型的生产系统
在NVIDIA的NVL72/576和华为的CloudMatrix384等高带宽超规模集群上部署稀疏专家混合（MoE）模型时，尽管具备统一全局地址空间和高带宽互联，仍面临三大核心瓶颈：(1) 粗粒度批量同步并行（BSP）导致执行严格串行；(2) 同步开销随高带宽增长而急剧上升；(3) 无距离感知调度引发不均的不规则令牌流量负载。为此，我们提出UBEP（统一总线专家并行），一种面向现代超规模架构的生产级通信库，重新设计了MoE的All-to-All原语。大规模实验表明，UBEP将All-to-All延迟降低最多52.4%，MoE推理每输出令牌时间（TPOT）减少最多11.1%。
原文摘要 · Abstract (English)
The deployment of Mixture-of-Experts (MoE) models on production high-bandwidth superpods, such as NVIDIA's NVL72/576 and Huawei's CloudMatrix384, introduces critical challenges beyond raw interconnect bandwidth. While these systems provide unified global address spaces and high-bandwidth fabrics, their full potential for sparse MoE communication is hindered by three fundamental bottlenecks: (1) Strict execution serialization imposed by coarse-grained Bulk Synchronous Parallel (BSP) orchestration of interdependent communication phases; (2) Prohibitive synchronization overhead that fails to scale alongside high interconnect bandwidth; and (3) Severe load imbalance resulting from distance-agnostic scheduling of irregular token traffic. To eliminate these bottlenecks, we introduce UBEP (Unified-Bus Expert Parallelism), a production-ready communication library that rethinks MoE's All-to-All primitives for modern superpod architectures. Through large scale experiments, UBEP reduces All-to-All latency by up to 52.4% and MoE inference Time Per Output Token (TPOT) by up to 11.1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。