提出新架构与并行方法，让大模型训练更快更稳
Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism
- 用多头潜在专家路由和头并行实现通信开销恒定
- 训练速度比传统方法快1.61倍，且负载均衡无延迟
- 适合大规模模型训练，尤其适合资源有限的研究者
大语言模型虽应用广泛，但训练成本高昂。稀疏专家混合（MoE）通过条件计算缓解此问题，当前主流分布式训练方法为专家并行（EP），但存在三大缺陷：通信开销随激活专家数k线性增长，负载不均影响延迟与内存，数据依赖通信需额外元信息交换。本文提出多头潜在专家混合与头并行（HP）架构与并行策略，实现与激活专家数k无关的O(1)通信开销、完全均衡的通信流量以及确定性通信，同时兼容现有EP方案。为加速多头潜在专家混合，设计了输入输出感知路由与专家计算优化。相比传统EP下的MoE，所提方法在保持性能一致的前提下，训练速度最高提升1.61倍；当粒度加倍时，整体性能更高，仍保持1.11倍加速。该方法显著降低了超大规模基础模型研究的门槛。
原文摘要 · Abstract (English)
Large language models have transformed many applications but remain expensive to train. Sparse Mixture of Experts (MoE) addresses this through conditional computation, with Expert Parallel (EP) as the standard distributed training method. However, EP has three limitations: communication cost grows linearly with the number of activated experts $k$, load imbalance affects latency and memory usage, and data-dependent communication requires metadata exchange. We propose Multi-Head LatentMoE and Head Parallel (HP), a new architecture and parallelism achieving $O(1)$ communication cost regardless of $k$, completely balanced traffic, and deterministic communication, all while remaining compatible with EP. To accelerate Multi-Head LatentMoE, we propose IO-aware routing and expert computation. Compared to MoE with EP, Multi-Head LatentMoE with HP trains up to $1.61\times$ faster while having identical performance. With doubled granularity, it achieves higher overall performance while still being $1.11\times$ faster. Our method makes multi-billion-parameter foundation model research more accessible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。