提出自适应专家剪枝方法，让大模型更省资源且不丢性能。
DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
- 按层动态调整剪枝率，捕捉不同层专家冗余差异。
- 在Mixtral 8×7B上仅用一半专家，保留92%原性能。
- 适合追求高效推理的大模型部署者。
尽管混合专家（MoE）模型取得显著突破，其规模增长也带来了巨大的内存与存储挑战。现有剪枝方法对各层采用统一稀疏度，因不同层专家冗余程度不一，常导致性能下降。为此，我们提出一种非均匀剪枝策略——可微专家剪枝（DiEP），可在层级别自适应调整剪枝率，并联合学习层间重要性，有效捕捉不同层间的冗余差异。通过将全局离散搜索空间转化为连续空间，方法支持指数级增长的非均匀专家组合，实现基于梯度的自适应剪枝。在五种先进MoE模型上的实验表明，该方法在多种NLP任务中均具有效性。尤其在Mixtral 8×7B上，仅保留一半专家即可维持约92%原始性能，在挑战性MMLU数据集上相比其他剪枝方法最高提升7.1%。
原文摘要 · Abstract (English)
Despite the significant breakthrough of Mixture-of-Experts (MoE), the increasing scale of these MoE models presents huge memory and storage challenges. Existing MoE pruning methods, which involve reducing parameter size with a uniform sparsity across all layers, often lead to suboptimal outcomes and performance degradation due to varying expert redundancy in different MoE layers. To address this, we propose a non-uniform pruning strategy, dubbed \textbf{Di}fferentiable \textbf{E}xpert \textbf{P}runing (\textbf{DiEP}), which adaptively adjusts pruning rates at the layer level while jointly learning inter-layer importance, effectively capturing the varying redundancy across different MoE layers. By transforming the global discrete search space into a continuous one, our method handles exponentially growing non-uniform expert combinations, enabling adaptive gradient-based pruning. Extensive experiments on five advanced MoE models demonstrate the efficacy of our method across various NLP tasks. Notably, \textbf{DiEP} retains around 92\% of original performance on Mixtral 8$\times$7B with only half the experts, outperforming other pruning methods by up to 7.1\% on the challenging MMLU dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。