通过合并专家权重，在不删减专家的情况下降低MoE模型内存占用。
REAM: Merging Improves Pruning of Experts in LLMs
- 不删专家，而是按路由权重合并专家权重。
- 在多任务评测中，合并后性能接近原始模型。
- 可灵活调节数据混合比例以平衡问答与生成能力。
Mixture-of-Experts（MoE）大语言模型是当前表现最优的架构之一。最大规模的模型通常拥有数百亿参数，部署时面临严重内存挑战。传统降低内存需求的方法包括权重剪枝和量化。受路由加权专家激活剪枝（REAP）的启发，我们提出一种新方法：路由加权专家激活合并（REAM）。不同于删除专家，REAM将专家分组并合并其权重，更有效地保持原始性能。我们在多个MoE LLM上，针对多种选择题问答（MC）和生成式（GEN）基准测试评估了REAM，结果揭示了MC与GEN性能之间的权衡，该权衡取决于校准数据的混合比例。通过控制通用、数学和编程数据的混合比例，我们分析了这一权衡的帕累托前沿，并表明REAM通常优于基线方法，在许多情况下可媲美原始未压缩模型。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。