通过输出合并优化压缩MoE模型，提升效率与性能。
MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
- 从输出合并视角构建数学优化框架，替代传统参数聚合。
- 在多个MoE模型上实现相同压缩率下优于基线的表现。
- 适合关注大模型压缩与高效推理的研究者与工程师。
混合专家（MoE）技术已被证明是高效扩展模型规模的有力方案，广泛应用于近期大语言模型的发展中。然而，MoE模型的显著内存开销使其压缩成为重要研究方向。本文对专家合并这一新兴压缩技术进行了理论分析，不再从参数聚合的传统视角理解，而是从专家输出合并的角度出发。核心洞察是：合并过程可被解释为在前向计算中插入额外矩阵，从而自然导出一个优化公式。基于此分析，我们提出MergeMoE方法，利用数学优化构建压缩矩阵。在多个MoE模型上的评估表明，该算法在相同压缩比下始终优于现有基线。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However, the substantial memory overhead of MoE models has made their compression an important research direction. In this work, we provide a theoretical analysis of expert merging, a recently proposed technique for compressing MoE models. Rather than interpreting expert merging from the conventional perspective of parameter aggregation, we approach it from the perspective of merging experts' outputs. Our key insight is that the merging process can be interpreted as inserting additional matrices into the forward computation, which naturally leads to an optimization formulation. Building on this analysis, we introduce MergeMoE, a method that leverages mathematical optimization to construct the compression matrices. We evaluate MergeMoE on multiple MoE models and show that our algorithm consistently outperforms the baselines with the same compression ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。