提出分层剪枝框架,让专家模型跨领域通用,减少重剪枝成本。
Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
- 通过聚类-选代表的结构化流程,按任务相似性整合专家
- 在通用任务上提升7.24%,专业任务上提升8.92%
- 适合需要多场景部署的大型专家模型应用
稀疏混合专家(SMoE)架构通过推理时仅激活部分参数,显著提升大语言模型性能。但其实际部署受限于所有专家需常驻内存带来的巨大静态内存开销。现有训练后剪枝方法通常基于单一通用语料库制定剪枝标准,导致在新领域上性能急剧下降,需为每个新领域重新剪枝,成本高昂。为此,我们提出Mosaic Pruning(MoP)。其核心思想是通过‘聚类-选取’的结构化过程构建功能全面的专家集合:利用跨任务表现相似性度量对专家进行功能聚类,并基于提出的激活可变性得分从每类中选出最具代表性专家。与仅优化单一语料库的方法不同,MoP确保剪枝后模型保留功能互补的专家组合,如同拼图般完整覆盖原模型能力,实现对多样化下游任务的泛化处理。在多个MoE模型上的大量实验表明,该方法显著优于现有工作,在通用任务上提升7.24%,在数学推理与代码生成等专用任务上提升8.92%。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (SMoE) architectures have enabled a new frontier in scaling Large Language Models (LLMs), offering superior performance by activating only a fraction of their total parameters during inference. However, their practical deployment is severely hampered by substantial static memory overhead, as all experts must be loaded into memory. Existing post-training pruning methods, while reducing model size, often derive their pruning criteria from a single, general-purpose corpus. This leads to a critical limitation: a catastrophic performance degradation when the pruned model is applied to other domains, necessitating a costly re-pruning for each new domain. To address this generalization gap, we introduce Mosaic Pruning (MoP). The core idea of MoP is to construct a functionally comprehensive set of experts through a structured ``cluster-then-select" process. This process leverages a similarity metric that captures expert performance across different task domains to functionally cluster the experts, and subsequently selects the most representative expert from each cluster based on our proposed Activation Variability Score. Unlike methods that optimize for a single corpus, our proposed Mosaic Pruning ensures that the pruned model retains a functionally complementary set of experts, much like the tiles of a mosaic that together form a complete picture of the original model's capabilities, enabling it to handle diverse downstream tasks.Extensive experiments on various MoE models demonstrate the superiority of our approach. MoP significantly outperforms prior work, achieving a 7.24\% gain on general tasks and 8.92\% on specialized tasks like math reasoning and code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。