将线性序列建模与专家混合结合，实现高效大模型训练。
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
- 融合线性序列建模与专家混合，支持线性复杂度建模。
- 在0.3B至7B参数模型上保持性能的同时提升训练效率。
- 专为大规模模型设计，适合追求高效推理的工程应用。
线性序列建模（LSM）如线性注意力、状态空间模型和线性RNN，以及专家混合（MoE）近期成为重要架构改进。本文提出Linear-MoE，一个用于大规模模型建模与训练的生产级系统，整合了LSM与MoE。Linear-MoE结合了LSM的线性复杂度序列建模优势与MoE的稀疏激活特性，旨在实现高性能与高效训练。系统包含：1）建模子系统，提供统一框架支持所有LSM实例；2）训练子系统，通过多种先进并行技术（尤其是针对Linear-MoE设计的序列并行）实现高效训练。此外，探索了将Linear-MoE层与标准Transformer-MoE层结合的混合模型，并引入序列并行以增强灵活性与性能。在A0.3B-2B和A1B-7B两个模型系列上的评估表明，Linear-MoE在保持各类基准上竞争力的同时实现效率提升，展现出作为下一代基础模型架构的潜力。代码已开源：https://github.com/OpenSparseLLMs/Linear-MoE。
原文摘要 · Abstract (English)
Linear Sequence Modeling (LSM) like linear attention, state space models and linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant architectural improvements. In this paper, we introduce Linear-MoE, a production-level system for modeling and training large-scale models that integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules for linear-complexity sequence modeling and MoE layers for sparsely activation, aiming to offer high performance with efficient training. The Linear-MoE system comprises: 1) Modeling subsystem, which provides a unified framework supporting all instances of LSM. and 2) Training subsystem, which facilitates efficient training by incorporating various advanced parallelism technologies, particularly Sequence Parallelism designed for Linear-MoE models. Additionally, we explore hybrid models that combine Linear-MoE layers with standard Transformer-MoE layers with its Sequence Parallelism to further enhance model flexibility and performance. Evaluations on two model series, A0.3B-2B and A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining competitive performance on various benchmarks, showcasing its potential as a next-generation foundational model architecture. Code: https://github.com/OpenSparseLLMs/Linear-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。