X-MoE让大规模专家模型在非NVIDIA超算上高效训练，突破5000亿参数瓶颈。
X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
- 无填充的跨平台专家路由，减少内存开销
- 1024张AMD显卡上训练5450亿参数模型，比现有方法大10倍
- 适合想在国产/异构超算上训练大MoE的团队
新兴的专家专用混合专家（MoE）架构，如DeepSeek-MoE，通过细粒度专家划分和大top-k路由实现优异模型性能。然而，其可扩展性受限于巨大的激活内存开销和昂贵的all-to-all通信。此外，现有MoE训练系统主要针对NVIDIA GPU优化，在非NVIDIA平台上表现不佳，导致计算潜力未被充分挖掘。本文提出X-MoE，一种新型MoE训练系统，可为下一代MoE架构提供可扩展训练性能。X-MoE通过多种创新技术实现：无填充的跨平台内核训练、冗余绕过分发机制，以及序列分片的混合并行策略。我们在搭载AMD MI250X GPU的前沿超算上评估表明，X-MoE可在1024张GPU上将DeepSeek风格的MoE扩展至5450亿参数，是同等硬件预算下现有方法可训练最大模型的10倍，同时保持高训练吞吐量。X-MoE源代码已开源。
原文摘要 · Abstract (English)
Emerging expert-specialized Mixture-of-Experts (MoE) architectures, such as DeepSeek-MoE, deliver strong model quality through fine-grained expert segmentation and large top-k routing. However, their scalability is limited by substantial activation memory overhead and costly all-to-all communication. Furthermore, current MoE training systems - primarily optimized for NVIDIA GPUs - perform suboptimally on non-NVIDIA platforms, leaving significant computational potential untapped. In this work, we present X-MoE, a novel MoE training system designed to deliver scalable training performance for next-generation MoE architectures. X-MoE achieves this via several novel techniques, including efficient padding-free MoE training with cross-platform kernels, redundancy-bypassing dispatch, and hybrid parallelism with sequence-sharded MoE blocks. Our evaluation on the Frontier supercomputer, powered by AMD MI250X GPUs, shows that X-MoE scales DeepSeek-style MoEs up to 545 billion parameters across 1024 GPUs - 10x larger than the largest trainable model with existing methods under the same hardware budget, while maintaining high training throughput. The source code of X-MoE is available at https://github.com/Supercomputing-System-AI-Lab/X-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。