轻量专家替换技术让大模型更省内存且不丢性能
LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
- 用高效模块替换冗余专家，降低存储开销
- 30%压缩率下性能媲美LoRA微调，50%压缩率提升5.6%
- 适合追求低内存高效率的模型部署场景
基于混合专家（MoE）的大语言模型展现出优异性能与计算效率，但其部署常受限于大量专家模块带来的内存压力。现有压缩方法如剪枝或合并往往导致不可逆的知识损失或高训练成本。本文提出一种新范式——专家替换，通过参数高效的模块替代冗余专家，并以低成本恢复其能力。实验表明，该基线已具良好表现。在此基础上，我们设计LightMoE框架，引入自适应专家选择、分层专家构建与渐进恢复策略。结果表明，在30%压缩率下，LightMoE性能可匹敌LoRA微调；在更激进的50%压缩率下，优于现有方法，在五个不同任务上平均提升5.6%。这证明LightMoE在内存效率、训练效率与模型性能间取得更优平衡。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by substantial memory demands, primarily due to the need to load numerous expert modules. While existing expert compression techniques like pruning or merging attempt to mitigate this, they often suffer from irreversible knowledge loss or high training overhead. In this paper, we propose a novel expert compression paradigm termed expert replacing, which replaces redundant experts with parameter-efficient modules and recovers their capabilities with low training costs. We find that even a straightforward baseline of this paradigm yields promising performance. Building on this foundation, we introduce LightMoE, a framework that enhances the paradigm by introducing adaptive expert selection, hierarchical expert construction, and an annealed recovery strategy. Experimental results show that LightMoE matches the performance of LoRA fine-tuning at a 30% compression ratio. Even under a more aggressive 50% compression rate, it outperforms existing methods and achieves average performance improvements of 5.6% across five diverse tasks. These findings demonstrate that LightMoE strikes a superior balance among memory efficiency, training efficiency, and model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。