不重训练就能合并稀疏专家模型，降低内存占用。
Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
- 基于专家输出的分层聚类，自动合并相似专家。
- 在Qwen和Mixtral上保持高精度，内存减少显著。
- 适合资源受限场景的LLM部署，无需重新训练。
稀疏混合专家（SMoE）模型通过高效参数利用推动大语言模型发展，在降低推理成本的同时实现性能提升。然而，专家组件的高内存需求限制了其在资源受限环境中的部署。本文提出无需重训练的分层聚类专家合并框架HC-SMoE，基于专家输出设计新型分层聚类方法，确保合并鲁棒性且独立于路由决策。该方法能有效捕捉大规模架构中专家间的功能关联。通过理论分析与多零样本语言任务的全面评估，验证了HC-SMoE在Qwen和Mixtral等先进模型上的有效性，结果表明其具备卓越性能与实际部署可行性。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (SMoE) models represent a significant advancement in large language model (LLM) development through their efficient parameter utilization. These models achieve substantial performance improvements at reduced inference costs. However, the deployment of SMoE models faces constraints from extensive memory requirements of expert components in resource-limited environments. To address these limitations, this paper introduces Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE), a task-agnostic expert merging framework for parameter reduction without retraining. HC-SMoE introduces a novel hierarchical clustering approach based on expert outputs to ensure merging robustness independent of routing decisions. The proposed output-based clustering method enables effective capture of functional relationships between experts for large-scale architectures. We provide theoretical analysis and comprehensive evaluations across multiple zero-shot language tasks to demonstrate HC-SMoE's effectiveness in state-of-the-art models including Qwen and Mixtral. The experimental results validate HC-SMoE's superior performance and practical applicability for real-world deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。