不重新训练就能压缩专家模型，提升推理效率。
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
- 通过路由器统计剪枝冗余专家，再按神经元级重组
- 在50%专家稀疏度下，常识推理与MMLU得分提升超5%
- 无需重训练，适合部署优化的大型专家混合模型
稀疏专家混合（SMoE）架构因计算高效被广泛用于大语言模型，但每个输入仍需加载全部专家参数，导致内存占用高，部署困难。现有工作多聚焦专家级剪枝，忽视神经元层级结构。本文提出DERN框架，无需重训练即可实现专家剪枝与重构。首先利用路由器统计剪除冗余专家；其次将专家分解为神经元级片段，分配给最兼容的保留专家；最后合并同一专家内的片段以构建紧凑表示。在Mixtral、Qwen和DeepSeek SMoE模型上的实验表明，当专家稀疏度达50%时，该方法在常识推理和MMLU基准上性能提升超过5%，同时显著减少专家数量与内存消耗，极大提升了模型部署可行性。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (SMoE) architectures are widely used in large language models (LLMs) due to their computational efficiency. However, though only a few experts are activated for each token, SMoE still requires loading all expert parameters, leading to high memory usage and challenges in deployment. Previous work has tried to reduce the overhead by pruning and merging experts, but primarily focused on expert-level operations, leaving neuron-level structure underexplored. We propose DERN (Dropping Experts, Recombining Neurons), a task-agnostic and retraining-free framework for expert pruning and reconstruction. We observe that experts are often misaligned and contain semantic conflicts at the neuron level, which poses challenges for direct merging. To solve this, DERN works in three steps: it first prunes redundant experts using router statistics; then it decomposes them into neuron-level expert segments, assigning each segment to its most compatible retained expert; and finally, it merges segments within each retained expert to build a compact representation. Experiments on Mixtral, Qwen, and DeepSeek SMoE models show that DERN improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity, without extra training. It also greatly reduces the number of experts and memory usage, making SMoE LLMs easier to deploy in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。