通过梯形结构剪枝专家神经元，实现大模型高效压缩且性能不降反升。
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
- 采用梯形结构剪枝，保留重要专家并精细化裁剪非关键神经元。
- 在40%路由专家稀疏下，准确率仅下降1点，代码生成提升10%。
- 适合追求高效率部署的LLM优化场景，尤其适合多专家架构。
基于稀疏激活的大规模混合专家语言模型（MoE LLM）虽可高效扩展，但其部署受限于专家的静态参数体量。现有压缩方法或直接移除完整专家，破坏路由结构并损害性能；或依赖非结构化权重剪枝，实际效率有限。为此，我们提出结构化的梯形专家神经元剪枝框架TENP。通过少量样本识别并保留重要专家，对次要专家实施专家神经元剪枝（ENP），将模型参数按从浅层到深层的梯形模式保留。评估专家重要性时，联合考虑专家输出幅度及其改变输入向量方向的能力；对于ENP，通过测量各神经元对专家输出的投影贡献来识别关键神经元。我们在Qwen和DeepSeek模型上进行大量实验，在路由专家稀疏度为40%、平均激活专家参数占比63.76%的情况下，DeepSeek模型相比全参数模型仅降低1分准确率，且在代码生成任务上反超10%。
原文摘要 · Abstract (English)
Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, reserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。