针对生成任务，剪枝比合并更有效压缩专家模型。
REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
- 基于路由权重与专家激活范数设计新剪枝准则
- 在50%压缩率下生成任务表现优于合并方法
- 代码生成任务可实现近乎无损压缩，适合大模型轻量化
稀疏激活的专家混合模型（SMoE）虽具备高效预训练和低延迟优势，但参数量庞大导致显著内存开销，推动专家压缩研究。与近期在判别任务中倾向专家合并的发现相反，我们发现生成任务中剪枝更具优势。原因在于现有合并技术会因丧失对专家的细粒度路由控制而引入不可消除的误差。基于此，我们提出路由器权重感知的专家激活剪枝（REAP），结合路由门值与专家激活范数，以最小化重建误差上界。在20B至1T参数的多种SMoE模型上，REAP在生成基准测试中持续优于合并及其他剪枝方法，尤其在50%压缩率时表现突出。值得注意的是，该方法在Qwen3-Coder-480B和Kimi-K2的代码生成任务中，即使剪除50%专家仍实现近乎无损压缩。
原文摘要 · Abstract (English)
Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。