针对专家模型训练通信瓶颈，提出智能并行策略提升效率。
MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimization
- 根据通信量和网络拓扑动态选择最优并行方案。
- 8卡张量并行下AllToAll性能提升8倍，2×70B模型训练延迟降13%。
- 适合大规模MoE模型训练，尤其在复杂网络环境下有优势。
混合专家（Mixture of Experts, MoE）是一种先进的工业级模型架构，通过整合多个领域专用的专家模型形成单一超模型，可在不显著增加训练与推理计算成本的前提下实现模型扩展，并最大化性能表现。然而，现有的分布式训练框架未充分优化通信效率，尤其对大型基础模型而言。本文提出一种网络流量感知的并行优化方法，依据通信量及训练集群的节点间与节点内网络拓扑结构，自动选择最优并行策略。实验表明，在8卡张量并行配置下，相较于DeepSpeed，MoNTA实现了8倍的AllToAll通信性能提升；在使用16块A800 GPU、序列长度为8K的情况下，训练2×70B模型相较基线整体延迟降低13%。
原文摘要 · Abstract (English)
The Mixture of Experts (MoE) is an advanced model architecture in the industry that combines multiple specialized expert models from various domains into a single supermodel. This approach enables the model to scale without significantly increasing the computational costs of training and inference, while maximizing model performance. However, current distributed training frameworks do not consider the ultimate optimization of communication, especially for large base models. This paper proposes a network-traffic-aware parallel optimization method that selects the optimal parallel strategy based on the communication volume, and the training cluster's inter-node and intra-node network topologies. Compared to the DeepSpeed, MoNTA achieves an 8x increase in AllToAll communication performance under 8-card tensor parallelism. Compared to the baseline, training a 2x70B model using 16 A800 cards, with an 8K sequence, results in a 13% overall latency performance improvement. Project Page: https://github.com/EnflameTechnology/DeepSpeed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。