arXiv:2606.09886cs.LGcs.AI2026-06被引 1

提出新方法,让稀疏专家模型更省显存且不丢性能。

SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs

论文配图:SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
图 1 · 摘自论文原文
  • 基于合作博弈思想,衡量专家间协作价值而非单独评分
  • 在3个主流MoE模型上实现20%~40%剪枝后仍保精度
  • 适合追求低内存部署的LLM研发者和工程师

稀疏混合专家(MoE)大模型虽以低每标记计算量实现高性能,但部署受限于内存墙:需常驻全部专家以支持动态路由。现有专家剪枝方法多独立评估专家,忽略MoE推理本质是组合协同——输出来自被选中的前k个专家组合。本文提出SHAPE,一种任务驱动的剪枝框架,显式建模层内专家协作。将小规模校准集上的路由轨迹视为经验合作博弈,通过类谢林值分配机制计算各专家在实际前k组合中的贡献,识别对高价值协作不可或缺的专家。为在全局剪枝预算下保持拓扑结构,引入“质量-覆盖”选择策略:每层保留最小专家子集,覆盖α比例的非负谢林值质量,同时用二分法匹配目标保留率。在三个现代MoE骨干模型(Qwen3-30B-A3B、GPT-OSS-20B、DeepSeek-V2-Lite)上跨多种基准测试验证,SHAPE显著优于全局与层级剪枝方法,在20%和40%专家剪枝下维持竞争力准确率,无需额外训练即实现明显峰值显存降低。开源代码见https://github.com/Alizen-1009/Shapley-Moe。

原文摘要 · Abstract (English)

Sparse Mixture-of-Experts (MoE) large language models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall: the full expert pool must remain resident to support token-dependent routing. Expert pruning is a direct remedy, but prior criteria typically score experts independently and overlook that MoE inference is inherently \emph{coalitional}, where outputs arise from routed top-$k$ expert combinations. We propose \textbf{SHAPE}, a task-driven pruning framework that explicitly models \emph{intra-layer} expert cooperation. SHAPE formulates routing traces on a small calibration set as an empirical cooperative game and assigns interaction-aware expert values via a Shapley-style attribution over observed top-$k$ coalitions, enabling the identification of experts that are essential for high-utility collaborations rather than merely frequent. To preserve MoE topology under a global pruning budget, SHAPE further introduces a \emph{quality-coverage} selection rule that retains, in each layer, the minimal expert subset covering an $α$ fraction of non-negative Shapley mass, while using bisection to match a target keep rate. Experiments on three modern MoE backbones (Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite) across diverse benchmarks show that SHAPE consistently improves robustness over global and layer-wise pruning variants, maintaining competitive accuracy under 20\% and 40\% expert pruning without additional training and delivering clear reductions in peak GPU memory footprint. The open-source code is available at https://github.com/Alizen-1009/Shapley-Moe.

专家模型模型剪枝稀疏性显存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。