用通用语料提升稀疏专家模型剪枝效果,不依赖下游数据。
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
- 分语料独立评估专家重要性,避免单一评分偏差。
- 在25%~75%保留率下,零样本任务准确率显著优于现有方法。
- 无需下游数据,适合大规模模型轻量化部署场景。
稀疏激活的混合专家(MoE)语言模型中存在大量结构化冗余,但无下游校准数据时进行剪枝仍具挑战。现有方法通常依赖单一聚合重要性评分,易偏向主导校准模式的专家。本文提出通用TB-覆盖率(Generic TB-Coverage)剪枝方法,仅使用通用语料(WikiText2和C4)进行校准。不将专家效用合并为单一指标,而是分别在各语料上独立评估专家性能,并施加固定预算覆盖率规则,在构建最终剪枝掩码前保留来自每种语料的高价值专家。在Qwen1.5-MoE-A2.7B与DeepSeek-MoE-16B-Base上,于25%、50%、75%保留率下,该方法在六个常见零样本基准上平均准确率优于随机剪枝、REAP与ExpertSparsity,同时降低在WikiText2和C4上的困惑度退化。在激进剪枝(25%和50%保留)时收益最大,表明跨语料专家覆盖是一种有效的通用先验。改进效果在固定剪枝预算下稳定,且无需下游校准数据。
原文摘要 · Abstract (English)
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose \textbf{Generic TB-Coverage}, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25\%, 50\%, and 75\% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25\% and 50\% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。