开源多语言稀疏专家模型，用5%参数实现高效训练与高精度
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
- 仅激活5%参数，通过密集模型迁移提升训练效率
- 在5万亿词上训练，性能超越同类模型且算力比最优
- 支持低成本扩展新语言，适合需要多语种推理的团队
我们提出Marco-MoE，一套完全开源的多语言稀疏专家（MoE）模型。其设计极为稀疏，每输入词元仅激活约5%的总参数。结合从密集模型迁移而来的上采样技术，可在5万亿词上高效预训练。模型在英文和多语言基准上表现超越同等规模竞争者，实现最佳性能-算力比。进一步微调得到Marco-MoE-Instruct变体，在仅激活3–14倍少参数的情况下仍优于其他模型。分析显示，Marco-MoE在相关语言间学习到共享的专家激活模式，同时对孤立语言保持高度专业化利用。还证明该模型可无干扰地实现可扩展的语言扩展。为支持社区，我们公开全部训练数据集、配方及模型权重。
原文摘要 · Abstract (English)
We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-\textsc{Instruct} variants, which surpass the performance of competing models possessing $3$--$14\times$ more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。