通过聚类剪枝，大幅压缩专家混合模型的参数量且保持性能。
Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
- 按功能相似性聚类专家，再全局剪除冗余集群。
- 在多个模型上实现更小体积，性能优于现有剪枝方法。
- 适合需要轻量化部署的MoE大模型应用者。
专家混合（MoE）架构已成为扩展大型语言模型（LLM）的有前景范式，通过任务特定专家的稀疏激活实现计算高效。尽管推理时效率高，但MoE模型整体参数量巨大（如GPT-4），给实际部署带来挑战。现有剪枝方法常忽视MoE系统的两个固有特性：1）层内专家同质性——同一层内专家功能冗余；2）层间相似性模式——深层专家趋于更同质。为此，我们提出聚类驱动的专家剪枝（C-Prune），一种两阶段自适应任务特定压缩框架。C-Prune通过层内专家聚类，利用参数相似性度量将功能相似的专家分组，再通过全局聚类剪枝，基于统一重要性评分机制消除跨层冗余集群，兼顾层间同质性。我们在多个MoE模型和基准上验证了C-Prune，结果表明其能有效降低模型规模，同时超越现有MoE剪枝方法。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have emerged as a promising paradigm for scaling large language models (LLMs) with sparse activation of task-specific experts. Despite their computational efficiency during inference, the massive overall parameter footprint of MoE models (e.g., GPT-4) introduces critical challenges for practical deployment. Current pruning approaches often fail to address two inherent characteristics of MoE systems: 1).intra-layer expert homogeneity where experts within the same MoE layer exhibit functional redundancy, and 2). inter-layer similarity patterns where deeper layers tend to contain progressively more homogeneous experts. To tackle these issues, we propose Cluster-driven Expert Pruning (C-Prune), a novel two-stage framework for adaptive task-specific compression of MoE LLMs. C-Prune operates through layer-wise expert clustering, which groups functionally similar experts within each MoE layer using parameter similarity metrics, followed by global cluster pruning, which eliminates redundant clusters across all layers through a unified importance scoring mechanism that accounts for cross-layer homogeneity. We validate C-Prune through extensive experiments on multiple MoE models and benchmarks. The results demonstrate that C-Prune effectively reduces model size while outperforming existing MoE pruning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。