通过路由提示剪枝专家模型，大幅减少冗余且无需重训练。
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
- 根据输入激活与路由权重乘积最小的权重进行剪枝。
- 50%稀疏度下，8x7B模型性能保留99%。
- 可结合教师模型知识蒸馏进一步提升剪枝后效果。
混合专家（MoE）架构面临内存消耗高和专家冗余的问题。剪枝可减少网络参数量并维持模型性能。受大语言模型中涌现的大梯度特征及MoE路由策略启发，我们提出MoE-Pruner，该方法基于每个输出神经元的最小权重乘积（权重×输入激活×路由权重）进行剪枝，为一次性剪枝，无需微调或权重更新。我们在Mixtral-8x7B和Mixtral-8x22B上评估该方法，在多个语言基准上表现显著优于现有LLM剪枝方法。实验表明，经过专家级知识蒸馏后，稀疏度达50%的Mixtral-8x7B模型仍能保持原始模型99%的性能。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model performance. Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot, requiring no retraining or weight updates. We evaluate our method on Mixtral-8x7B and Mixtral-8x22B across multiple language benchmarks. Experimental results show that our pruning method significantly outperforms state-of-the-art LLM pruning methods. Furthermore, our pruned MoE models can benefit from a pretrained teacher model through expert-wise knowledge distillation, improving performance post-pruning. Experimental results demonstrate that the Mixtral-8x7B model with 50% sparsity maintains 99% of the performance of the original model after the expert-wise knowledge distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。