提出全局感知的专家剪枝方法，提升稀疏MoE模型效率与性能。
Does a Global Perspective Help Prune Sparse MoEs Elegantly?
- 基于跨层冗余动态分配剪枝预算，突破传统均匀剪枝局限。
- 在相同剪枝预算下，平均性能优于最强局部基线1.40%，最高提升2.45%。
- 适用于大模型优化，尤其适合追求高效部署的科研与工程团队。
语言模型的实证缩放定律推动了更大规模LLM的发展，但其计算与内存开销日益严峻。稀疏混合专家（MoE）通过每前向传播仅激活部分专家，提升了效率且不牺牲性能。然而，专家参数量庞大仍导致显著内存消耗。现有剪枝方法通常在各层均匀分配剪枝预算，忽视了稀疏MoE中固有的异质冗余。本文提出GRAPE（Global Redundancy-Aware Pruning of Experts），一种基于跨层冗余动态分配剪枝预算的全局剪枝策略。在Mixtral-8x7B、Mixtral-8x22B、DeepSeek-MoE、Qwen-MoE和GPT-OSS上的实验表明，相同剪枝预算下，GRAPE始终实现最佳平均性能。在论文报告的三个主模型上，其平均准确率较最强局部基线提升1.40%，最高达2.45%。
原文摘要 · Abstract (English)
Empirical scaling laws for language models have encouraged the development of ever-larger LLMs, despite their growing computational and memory costs. Sparse Mixture-of-Experts (MoEs) offer a promising alternative by activating only a subset of experts per forward pass, improving efficiency without sacrificing performance. However, the large number of expert parameters still leads to substantial memory consumption. Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs. We propose GRAPE (Global Redundancy-Aware Pruning of Experts, a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy. Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance. On the three main models reported in the paper, it improves average accuracy over the strongest local baseline by 1.40% on average across pruning settings, with gains of up to 2.45%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。