用稀疏专家模型解析LLaMA的潜力，提升大模型效率。
LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
- 将注意力与MLP模块改造成稀疏专家结构，研究不同专家设计效果。
- 在保持激活参数不变的前提下，实现模型规模扩展，性能损失可控。
- 适合关注大模型压缩、高效训练与推理的研究者和工程师。
受稀疏性启发，混合专家（MoE）模型因其在保持激活参数数量不变的同时扩展模型规模而日益流行。本研究通过为Transformer块中的注意力（即注意力MoE）和MLP（即MLP MoE）模块构建MoE结构，深入探究密集型LLaMA模型的稀疏性。我们分析了在相同激活条件下不同专家构建方法与粒度的影响。为进一步评估稀疏化后模型在对话、代码、数学等多领域的能力，将稀疏性应用于指令微调的大语言模型，并构建指令型MoE模型。为缓解稀疏性带来的性能下降，设计了两阶段后训练策略以提升模型表现。在LLaMA3上的实验表明该方法对未来指令型MoE模型的发展具有潜在有效性。源代码与模型见： https://github.com/OpenSparseLLMs/LLaMA-MoE-v2。
原文摘要 · Abstract (English)
Recently, inspired by the concept of sparsity, Mixture-of-Experts (MoE) models have gained increasing popularity for scaling model size while keeping the number of activated parameters constant. In this study, we thoroughly investigate the sparsity of the dense LLaMA model by constructing MoE for both the attention (i.e., Attention MoE) and MLP (i.e., MLP MoE) modules in the transformer blocks. Specifically, we investigate different expert construction methods and granularities under the same activation conditions to analyze the impact of sparsifying the model. Additionally, to comprehensively evaluate the model's capabilities across various domains (e.g., conversation, code, math) after sparsification, we apply sparsity to the instructed large language models (LLMs) and construct instructed MoE models. To counteract the performance degradation resulting from increased sparsity, we design a two-stage post-training strategy to enhance model performance. Experiments on the LLaMA3 model demonstrate the potential effectiveness of this approach for future developments of instructed MoE models. The source codes and models are available at: \url{https://github.com/OpenSparseLLMs/LLaMA-MoE-v2}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。