让每个专家都出力，提升MoE模型压缩效果
Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
- 通过多轮采样和路由优化，挖掘非激活专家的知识
- 在多个数据集上超越传统蒸馏方法，显著提升学生模型性能
- 适合需要高效部署MoE大模型的开发者与研究者
随着混合专家（Mixture-of-Experts, MoE）模型的兴起，大语言模型的高效扩展成为可能。然而，其高内存需求限制了在资源受限环境中的应用。尽管知识蒸馏（KD）是模型压缩的有效手段，但其在MoE教师模型上的应用仍不充分。我们发现，MoE模型中未激活的专家也蕴含重要知识，而现有蒸馏方法未能有效利用这些知识。为此，我们首次提出两种面向MoE的蒸馏方法：知识增强（KA）与学生感知路由（SAR）。KA通过多次采样专家来扩充知识，SAR则利用所有专家并通过路由训练调整权重以提供最优知识。大量实验表明，我们的方法显著优于传统蒸馏方法，验证了其在压缩MoE模型上的有效性。
原文摘要 · Abstract (English)
With the emergence of Mixture-of-Experts (MoE), the efficient scaling of model size has accelerated the development of large language models in recent years. However, their high memory requirements prevent their use in resource-constrained environments. While knowledge distillation (KD) has been a proven method for model compression, its application to MoE teacher models remains underexplored. Through our investigation, we discover that non-activated experts in MoE models possess valuable knowledge that benefits student models. We further demonstrate that existing KD methods are not optimal for compressing MoE models, as they fail to leverage this knowledge effectively. To address this, we propose two intuitive MoE-specific KD methods for the first time: Knowledge Augmentation (KA) and Student-Aware Router (SAR), both designed to effectively extract knowledge from all experts. Specifically, KA augments knowledge by sampling experts multiple times, while SAR uses all experts and adjusts the expert weights through router training to provide optimal knowledge. Extensive experiments show that our methods outperform conventional KD methods, demonstrating their effectiveness for MoE teacher models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。