通过分解专家权重，实现大模型高效压缩且无需重新训练。
Delta Decompression for MoE-based LLMs Compression
- 将专家权重拆分为共享基底和差异增量，利用信息矩阵融合共性。
- 在40%~60%压缩率下，性能优于现有方法超13%。
- 适合需要轻量化部署的MoE类大模型开发者使用。
基于混合专家（MoE）架构的大语言模型虽表现优异，但存储与内存开销巨大。为此，我们提出D²-MoE，一种用于压缩MoE大模型参数的新型增量解压方法。基于专家权重的多样性观察，我们将权重分解为共享基底与独特增量两部分：首先利用Fisher信息矩阵将各专家权重合并至基底以捕捉共性；随后通过奇异值分解（SVD）压缩增量部分，利用其低秩特性；最后引入半动态结构化剪枝策略对基底权重进行压缩，结合静态与动态冗余分析，在保持输入适应性的同时进一步减少参数量。该方法可在不需额外训练的情况下实现高比例压缩。大量实验表明，在Mixtral、Phi-3.5、DeepSeek、Qwen2等MoE模型上，40%~60%压缩率下性能相较其他压缩器提升超过13%。代码已开源于https://github.com/lliai/D2MoE。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures in large language models (LLMs) achieve exceptional performance, but face prohibitive storage and memory requirements. To address these challenges, we present $D^2$-MoE, a new delta decompression compressor for reducing the parameters of MoE LLMs. Based on observations of expert diversity, we decompose their weights into a shared base weight and unique delta weights. Specifically, our method first merges each expert's weight into the base weight using the Fisher information matrix to capture shared components. Then, we compress delta weights through Singular Value Decomposition (SVD) by exploiting their low-rank properties. Finally, we introduce a semi-dynamical structured pruning strategy for the base weights, combining static and dynamic redundancy analysis to achieve further parameter reduction while maintaining input adaptivity. In this way, our $D^2$-MoE successfully compact MoE LLMs to high compression ratios without additional training. Extensive experiments highlight the superiority of our approach, with over 13% performance gains than other compressors on Mixtral|Phi-3.5|DeepSeek|Qwen2 MoE LLMs at 40$\sim$60% compression rates. Codes are available in https://github.com/lliai/D2MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。