通过分层重构让专家模型更省通信，训练更快且性能不降。
Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
- 将专家层与混合层分离设计，减少通信开销
- 315亿参数下节省33.3%训练耗时，推理速度更快
- 适合大规模语言模型训练，尤其关注效率的团队
在使用专家并行训练混合专家（MoE）语言模型时，全对全的令牌分发与聚合操作会占用大量训练时间。本文提出通信高效型MoE模型（CE-MoE），采用异构层结构，解耦令牌混合与通道混合深度。相较于传统模型在每个令牌混合层后交替插入MoE层（如注意力或Mamba-2），CE-MoE将专家容量集中于少数路由层，同时通过增加额外的令牌混合层和密集前馈层维持模型深度。在从20亿到315亿总参数的扩展尺度上，在匹配总参数量和激活参数量的前提下，CE-MoE模型持续降低训练成本，同时保持与全MoE基线相当的验证损失和下游任务表现。在315亿参数规模下，CE-MoE节省了33.3%的GPU小时数，且平均下游得分和推理吞吐率均有所提升。
原文摘要 · Abstract (English)
When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3\% fewer GPU-hours while improving average downstream score and inference throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。