揭示大模型专家间的协作规律并优化剪枝，提升效率与可解释性
Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
- 提出分层稀疏字典学习法，发现专家协作有语义模式
- 通过贡献感知剪枝，平均性能提升2.5%，优于现有方法
- 适合研究模型可解释性与高效部署的AI从业者
基于专家混合（MoE）的大语言模型在多任务适应性方面表现突出，通过动态路由将输入分配给专用专家。尽管如此，专家间的协作机制仍不清晰，限制了模型的可解释性与优化空间。本文聚焦两大问题：(1) 识别专家协作模式；(2) 通过专家剪枝优化模型。针对前者，提出分层稀疏字典学习（HSDL）方法，揭示专家间协作模式；针对后者，设计贡献感知专家剪枝（CAEP）算法，有效移除低贡献专家。大量实验表明，专家协作模式与特定输入类型密切相关，并在多个任务中具有语义意义。剪枝实验显示，该方法平均性能提升2.5%，优于现有方法。研究为提升MoE大模型的效率与可解释性提供了新思路。
原文摘要 · Abstract (English)
Mixture-of-Experts based large language models (MoE LLMs) have shown significant promise in multitask adaptability by dynamically routing inputs to specialized experts. Despite their success, the collaborative mechanisms among experts are still not well understood, limiting both the interpretability and optimization of these models. In this paper, we focus on two critical issues: (1) identifying expert collaboration patterns, and (2) optimizing MoE LLMs through expert pruning. To address the first issue, we propose a hierarchical sparse dictionary learning (HSDL) method that uncovers the collaboration patterns among experts. For the second issue, we introduce the Contribution-Aware Expert Pruning (CAEP) algorithm, which effectively prunes low-contribution experts. Our extensive experiments demonstrate that expert collaboration patterns are closely linked to specific input types and exhibit semantic significance across various tasks. Moreover, pruning experiments show that our approach improves overall performance by 2.5\% on average, outperforming existing methods. These findings offer valuable insights into enhancing the efficiency and interpretability of MoE LLMs, offering a clearer understanding of expert interactions and improving model optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。