针对MoE大模型推理内存高、加速慢的问题，提出新型压缩方法。
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
- 通过校准路由器缓解低比特量化带来的专家选择偏差
- 按专家激活频率剪枝，显著降低推理延迟
- 适合部署在资源受限环境下的MoE大模型
Mixture-of-Experts（MoE）在扩展大语言模型方面展现出巨大潜力，但面临两大挑战：(1) 需加载所有专家导致大量GPU内存消耗；(2) 激活参数少无法等效转化为推理加速。本文提出EAC-MoE，一种面向专家选择的压缩方法，从量化与剪枝角度深度适配MoE特性，引入两个模块分别解决上述问题：(1) 低比特量化引起的专家选择偏差是导致性能下降的主要因素。为此提出量化校准（QESC），通过校准MoE中的路由器缓解该偏差；(2) 某些专家对当前任务不重要却增加推理延迟。为此提出基于专家激活频率的剪枝（PESF），通过剪除不常激活的专家显著提升推理速度。大量实验表明，该方法在极小性能损失下显著降低内存占用并提升推理效率。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs. However, it is hindered by two critical challenges: (1) substantial GPU memory consumption to load all experts; (2) low activated parameters cannot be equivalently translated into inference acceleration effects. In this work, we propose EAC-MoE, an Expert-Selection Aware Compressor for MoE-LLMs, which deeply aligns with the characteristics of MoE from the perspectives of quantization and pruning, and introduces two modules to address these two challenges respectively: (1) The expert selection bias caused by low-bit quantization is a major factor contributing to the performance degradation in MoE-LLMs. Based on this, we propose Quantization with Expert-Selection Calibration (QESC), which mitigates the expert selection bias by calibrating the routers within the MoE; (2) There are always certain experts that are not crucial for the corresponding tasks, yet causing inference latency. Therefore, we propose Pruning based on Expert-Selection Frequency (PESF), which significantly improves inference speed by pruning less frequently used experts for current task. Extensive experiments demonstrate that our approach significantly reduces memory usage and improves inference speed with minimal performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。