arXiv:2603.06003cs.LG2026-03被引 5

通过非均匀分配专家稀疏度,提升稀疏专家模型的生成性能。

EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

  • 用启发式搜索优化各层专家数量分配,不依赖固定稀疏比例。
  • 在25%~50%稀疏下,数学推理能力最高提升19.6%。
  • 无需重训练,可直接适配现有剪枝方法,适合部署优化。

稀疏专家混合(SMoE)语言模型在低每词计算量下具备强能力,但部署受限于内存占用和吞吐量,因需存储并服务完整专家池。后训练专家剪枝可降低开销,但多数方法仅关注每层内剪枝对象,采用均匀层间稀疏分配,而实际层间分配对性能影响显著。本文将剪枝解耦为层内专家排序与层间预算分配,并提出期望推测接受代理(ESAP),一种受推测解码启发、教师强制的指标,可在无成本自回归解码下衡量剪枝模型与全模型匹配程度。ESAP有界且稳定,支持低成本候选方案比较。基于此,提出EvoESAP进化搜索框架,在固定全局预算下寻找更优非均匀层间稀疏分配,同时保持层内剪枝顺序不变,可作为插件化方法适配频率、EAN、SEER、REAP等标准。在7B–30B规模的SMoE LLM上,于25%和50%稀疏度下,EvoESAP持续发现更优非均匀分配,显著提升开放式生成表现(如50%稀疏下MATH-500最高提升19.6%),同时维持与均匀剪枝相当的多项选择准确率。代码已开源。

原文摘要 · Abstract (English)

Sparse Mixture-of-Experts (SMoE) language models achieve strong capability at low per-token compute, yet deployment remains constrained by memory footprint and throughput because the full expert pool must still be stored and served. Post-training expert pruning reduces this cost, but most methods focus on which experts to prune within each layer and default to a uniform layer-wise sparsity allocation, even though the layer-wise allocation can strongly affect performance. We decouple pruning into within-layer expert ranking and across-layer budget allocation, and introduce \textbf{E}xpected \textbf{S}peculative \textbf{A}cceptance \textbf{P}roxy (\textbf{ESAP}), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model without costly autoregressive decoding. ESAP is bounded and stable, enabling cheap comparison of many candidates. Building on ESAP, we propose EvoESAP, an evolutionary search framework that finds an improved non-uniform layer-wise sparsity allocation under a fixed global budget while holding the within-layer pruning order fixed, making it a plug-and-play method for criteria such as Frequency, EAN, SEER, and REAP. Across 7B--30B SMoE LLMs at 25\% and 50\% sparsity, EvoESAP consistently discovers non-uniform allocations that improve open-ended generation (up to \textbf{+19.6\%} on MATH-500 at 50\% sparsity) while preserving competitive multiple-choice accuracy compared with uniform pruning at the same sparsity. Code is available at https://github.com/ZongfangLiu/EvoESAP.

稀疏模型专家混合剪枝优化生成性能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。