用少量计算量将Llama 3转为高效MoE模型，性能提升2%。
Llama 3 Meets MoE: Efficient Upcycling
- 基于预训练权重微调8专家混合模型，仅需1%算力
- 0样本测试在MMLU上准确率提升2%，训练时MFU达46.8%
- 支持在线升级，适合资源有限的模型扩容场景
扩展大型语言模型（LLMs）能显著提升性能，但计算成本高昂。混合专家（MoE）模型提供了一种高效替代方案，在不按比例增加计算需求的情况下提升模型容量。然而，从头训练MoE模型存在过拟合和路由不稳定等挑战。本文提出一种高效训练方法，利用预训练的密集模型检查点，仅需典型预训练算力的1%以下，即可从Llama 3-8B训练出一个8专家Top-2 MoE模型。该方法在学术基准上提升下游性能，0样本准确率在MMLU上提高2%，训练期间模型浮点运算利用率（MFU）达到46.8%。我们还集成NeMo中的在线升级机制，实现预训练权重的无缝复用，支持低成本构建高容量MoE模型。
原文摘要 · Abstract (English)
Scaling large language models (LLMs) significantly improves performance but comes with prohibitive computational costs. Mixture-of-Experts (MoE) models offer an efficient alternative, increasing capacity without a proportional rise in compute requirements. However, training MoE models from scratch poses challenges like overfitting and routing instability. We present an efficient training recipe leveraging pre-trained dense checkpoints, training an 8-Expert Top-2 MoE model from Llama 3-8B with less than $1\%$ of typical pre-training compute. Our approach enhances downstream performance on academic benchmarks, achieving a $\textbf{2%}$ improvement in 0-shot accuracy on MMLU, while reaching a Model FLOPs Utilization (MFU) of $\textbf{46.8%}$ during training using our framework. We also integrate online upcycling in NeMo for seamless use of pre-trained weights, enabling cost-effective development of high-capacity MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。