通过专家间剪枝与专家内低秩分解，高效压缩MoE大模型。
MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
- 分层遗传搜索+块级感受野分析，实现非均匀专家剪枝。
- 在保持零样本性能前提下，模型体积减少40%以上，推理速度提升2倍。
- 适合需要轻量化部署MoE模型的研究者与工程师。
Mixture of Experts（MoE）大语言模型的出现显著推动了语言模型的发展。相比传统LLM，MoE模型在激活参数极少的情况下仍能实现更高性能。然而其庞大的参数量仍导致高部署成本。本文提出一种两阶段压缩方法：首先在专家间剪枝阶段，通过分层遗传搜索与块级KT感受野分析，结合非均匀剪枝率，对各层专家进行重要性评估并剪枝；其次在专家内分解阶段，对保留专家内的参数实施低秩分解以进一步压缩。在Qwen1.5-MoE-A2.7B、DeepSeek-V2-Lite和Mixtral-8×7B上的大量实验表明，该方法可在保持多种零样本任务性能的前提下，有效减小模型规模并提升推理效率。代码将开源于https://github.com/xiaochengsky/MoEI-2.git。
原文摘要 · Abstract (English)
The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with considerably fewer activated parameters. Despite this efficiency, their enormous parameter size still leads to high deployment costs. In this paper, we introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. First, in the inter-expert pruning stage, we analyze the importance of each layer and propose the Layer-wise Genetic Search and Block-wise KT-Reception Field with the non-uniform pruning ratio to prune the individual expert. Second, in the intra-expert decomposition stage, we apply the low-rank decomposition to further compress the parameters within the remaining experts. Extensive experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8$\times$7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks. The code will be available at \url{https://github.com/xiaochengsky/MoEI-2.git}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。