用少量示例就能精准剪掉大模型中冗余专家,提升效率不丢性能。
Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
- 通过少量领域样本定位关键专家,只保留任务相关部分。
- 在相同内存下,推理速度提升2.99倍,性能几乎不变。
- 适合资源受限场景下的大模型高效部署,尤其适用于领域专用任务。
Mixture-of-Experts(MoE)模型通过仅激活部分专家,在性能与推理效率间取得良好平衡。然而,存储所有专家带来的内存开销仍是大规模MoE模型(如DeepSeek-R1(671B))的主要瓶颈。本研究探究了大规模MoE模型中的领域特异性和专家冗余性,发现一种称为少样本专家定位的现象:仅需少量领域内示例,模型在同领域任务上始终激活一组稀疏且稳定的专家子集。基于此观察,我们提出简单高效的剪枝框架EASY-EP,利用少量领域特定示例识别并保留最相关的专家。EASY-EP包含两个核心组件:输出感知的专家重要性评估和专家级令牌贡献估计。前者通过门控分数与激活专家输出的L2范数评估每个专家对当前令牌的重要性;后者根据路由专家前后表示的相似性评估令牌贡献。在DeepSeek-R1和DeepSeek-V3-0324上的实验表明,该方法可在仅保留一半专家的情况下,实现与全模型相当的性能,并在相同内存预算下获得2.99倍的吞吐量提升。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models achieve a favorable trade-off between performance and inference efficiency by activating only a subset of experts. However, the memory overhead of storing all experts remains a major limitation, especially in large-scale MoE models such as DeepSeek-R1(671B). In this study, we investigate domain specialization and expert redundancy in large-scale MoE models and uncover a consistent behavior we term few-shot expert localization, with only a few in-domain demonstrations, the model consistently activates a sparse and stable subset of experts on tasks within the same domain. Building on this observation, we propose a simple yet effective pruning framework, EASY-EP, that leverages a few domain-specific demonstrations to identify and retain only the most relevant experts. EASY-EP comprises two key components: output-aware expert importance assessment and expert-level token contribution estimation. The former evaluates the importance of each expert for the current token by considering the gating scores and L2 norm of the outputs of activated experts, while the latter assesses the contribution of tokens based on representation similarities before and after routed experts. Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99\times$ throughput under the same memory budget with full model with only half the experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。