用残差恢复压缩专家混合模型，大幅节省推理内存
ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
- 通过沃氏均值提取通用专家，用残差近似原专家
- 专家参数量最多减少75%，性能几乎不变
- 无需重训练，适合部署大模型的资源受限场景
专家混合（MoE）Transformer是多个顶尖语言模型的核心架构，通过仅激活部分参数实现稀疏性，虽保持恒定计算开销，却造成空间效率低下：推理时仍需加载全部参数。本文提出ResMoE，一种基于沃氏均值的新型MoE近似框架，通过提取一个公共专家（均值专家）并近似其与原专家间的残差，实现对专家的高效压缩。该方法在无需重新训练、数据无关的情况下，一次性提升大规模MoE Transformer的推理空间效率，同时保持极小精度损失，推动大模型更广泛的应用。我们在Switch Transformer、Mixtral和DeepSeekMoE上进行了大量实验，结果表明ResMoE可将单个专家参数量最多减少75%，且性能相当。代码已开源：https://github.com/iDEA-iSAIL-Lab-UIUC/ResMoE。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) Transformer, the backbone architecture of multiple phenomenal language models, leverages sparsity by activating only a fraction of model parameters for each input token. The sparse structure, while allowing constant time costs, results in space inefficiency: we still need to load all the model parameters during inference. We introduce ResMoE, an innovative MoE approximation framework that utilizes Wasserstein barycenter to extract a common expert (barycenter expert) and approximate the residuals between this barycenter expert and the original ones. ResMoE enhances the space efficiency for inference of large-scale MoE Transformers in a one-shot and data-agnostic manner without retraining while maintaining minimal accuracy loss, thereby paving the way for broader accessibility to large language models. We demonstrate the effectiveness of ResMoE through extensive experiments on Switch Transformer, Mixtral, and DeepSeekMoE models. The results show that ResMoE can reduce the number of parameters in an expert by up to 75% while maintaining comparable performance. The code is available at https://github.com/iDEA-iSAIL-Lab-UIUC/ResMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。