arXiv:2606.18304cs.LGcs.AI2026-06

针对MoE模型的冗余通道精准剪枝,提升压缩效率与精度。

Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression

论文配图:Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
图 1 · 摘自论文原文
  • 基于通道重要性分配剪枝比例,实现细粒度结构剪枝
  • 结合4比特量化,在50%或25%剪枝率下保持模型精度
  • 适合需要高效部署MoE模型的研究与工程人员

Mixture-of-Experts(MoE)模型虽能高效扩展计算量,但因内存占用大和推理开销高而难以部署。现有压缩方法多在专家层级操作,通过移除整个专家或依据粗粒度重要性评分排序,但此类决策过于粗糙,难以捕捉细粒度冗余,导致剪枝预算分配不当,压缩效果受限。我们观察到,即使在被判定为重要的专家中,信息也高度集中于少数通道,存在大量冗余。据此,提出一种面向MoE模型的结构化剪枝框架,将剪枝比例分配重构为通道得分覆盖率最大化问题,并采用基于归因的近似方法高效求解。在DeepSeek与Qwen MoE模型上的实验表明,该方法结合4比特量化后,在50%或25%结构化剪枝下仍可保持模型精度。在Qwen3-30B-A3B模型上,内存占用减少5.27倍,且在多个基准测试中持续优于现有最优基线。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead. Prior compression methods mainly operate at the expert level, either removing entire experts or ranking experts by coarse-grained importance scores. However, such expert-wise decisions are often too coarse to capture fine-grained redundancy, leading to misallocated pruning budgets and limited compression. To address this problem, we observe that information within MoE experts is highly concentrated in a small subset of channels, leaving substantial redundancy even in experts deemed important. Based on this observation, we propose a structural pruning framework tailored for MoE models. Our method reformulates prune-ratio allocation as a channel-score coverage maximization problem and solves it efficiently using an attribution-based approximation. Experiments on DeepSeek and Qwen MoE models show that our method preserves model accuracy under 50% or 25% structured pruning when combined with 4-bit quantization. On Qwen3-30B-A3B, our approach reduces memory footprint by 5.27$\times$ and consistently outperforms state-of-the-art baselines across diverse benchmarks.

MoE压缩结构剪枝4比特量化通道重要性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。