将大模型转为稀疏专家混合模型，提升性能且更高效。
Upcycling Large Language Models into Mixture of Experts
- 提出虚拟分组初始化与权重缩放，支持细粒度专家架构。
- 在1万亿token上训练后，性能从65.3%提升至67.6%。
- 适合追求高容量、低计算开销的大模型优化者。
将预训练的密集语言模型升级为稀疏专家混合（MoE）模型，是提升已有模型容量的有效方法。然而，大规模场景下的最优升级策略仍不明确。本文对数十亿参数级语言模型的升级方法与超参数进行了广泛研究。提出一种新的“虚拟分组”初始化方案和权重缩放方法，使模型可升级为细粒度MoE结构。通过消融实验发现，模型升级优于持续训练。此外，我们证明了‘Softmax-then-topK’路由优于‘topK-then-softmax’，且更高粒度的MoE有助于提升准确率。最终，在1万亿令牌上对Nemotron-4 15B进行升级，其在MMLU测试中达到67.6%，而同规模持续训练版本仅为65.3%。结果为构建高效MoE语言模型提供了实践指导。代码已开源。
原文摘要 · Abstract (English)
Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models. Code is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。