arXiv:2510.19470cs.DCcs.AI2025-10被引 4

提出HybridEP框架,解决跨数据中心专家并行通信瓶颈问题。

HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission

  • 动态调整专家分布以减少数据与专家传输频率
  • 在受限带宽下实现最高5.6倍的训练加速
  • 适合大规模跨数据中心模型训练场景

混合专家(MoE)已成为扩展大模型的流行架构。然而,模型规模迅速增长已超出单数据中心训练能力,推动向更灵活的跨数据中心训练范式转变。在此背景下,专家并行(EP)因跨数据中心带宽有限而面临显著可扩展性挑战。现有优化尝试重叠数据通信与计算,但在低带宽环境下因通信时间过长而效果有限。为此,我们提出HybridEP,一种基于建模的框架,在受限带宽下优化EP。核心思想是动态调整专家空间布局,以降低通信流量和频率,从而最小化通信开销。由于混合数据与专家通信使原通信模式复杂化,寻找最优解非易事。因此,我们构建了流式模型以确定最优传输比例。基于此,引入两项技术:(1) 域级划分,建立混合模式与GPU层级通信拓扑间的映射;(2) 参数高效迁移,通过减少专家传输开销并扩大域大小进一步优化拓扑。结合所有设计,HybridEP可视为更具可扩展性的通用型专家并行方案。实验表明,在受限带宽下,HybridEP相比现有最先进系统提升达5.6倍。大规模仿真对比显示,当使用1000个数据中心时,不同带宽下最高可获得1.45倍加速。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) has become a popular architecture for scaling large models. However, the rapidly growing scale outpaces model training on a single DC, driving a shift toward a more flexible, cross-DC training paradigm. Under this, Expert Parallelism (EP) of MoE faces significant scalability issues due to the limited cross-DC bandwidth. Specifically, existing EP optimizations attempt to overlap data communication and computation, which has little benefit in low-bandwidth scenarios due to a much longer data communication time. Therefore, the trends of cross-DC EP scaling is fast becoming a critical roadblock to the continued growth of MoE models. To address this, we propose HybridEP, a modeling-guided framework to optimize EP under constrained bandwidth. Our key idea is to dynamically transform the spatial placement of experts to reduce data communication traffic and frequency, thereby minimizing EP's communication overheads. However, it is non-trivial to find the optimal solution because it complicates the original communication pattern by mixing data and expert communication. We therefore build a stream-based model to determine the optimal transmission ratio. Guided by this, we incorporate two techniques: (1) domain-based partition to construct the mapping between hybrid patterns and specific communication topology at GPU level, and (2) parameter-efficient migration to further refine this topology by reducing expert transmission overhead and enlarging the domain size. Combining all these designs, HybridEP can be considered as a more general EP with better scalability. Experimental results show that HybridEP outperforms existing state-of-the-art MoE training systems by up to 5.6x under constrained bandwidth. We further compare HybridEP and EP on large-scale simulations. HybridEP achieves up to 1.45x speedup with 1k DCs under different bandwidths.

专家并行跨数据中心模型训练通信优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。