通过稀疏合并与位打包，实现大模型专家系统的高效压缩与推理。
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
- 基于权重冗余和专业化分析，稀疏合并专家参数。
- 50%压缩率下准确率保持稳定，比现有方法高16.7%。
- 无需训练，适配GPU，显著提升推理速度。
混合专家（MoE）模型通过仅激活部分专家实现了语言模型的高效扩展，但其广泛部署受限于存储所有专家参数带来的高内存开销。现有方法虽尝试通过专家删除或合并降低开销，但在高压缩比下常导致性能下降。本文提出PuzzleMoE，一种无需训练的MoE压缩方法，包含两项关键创新：首先，通过双掩码捕捉共享与专家特异性参数，实现稀疏专家合并；其次，采用位打包编码，复用未充分利用的指数位，避免存储二值掩码和符号的开销，支持高效GPU推理。大量实验表明，PuzzleMoE可在50%压缩率下保持模型精度，且在MMLU任务上相比现有方法最高提升16.7%，推理速度最高达1.28倍加速。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies, yet they often suffer from performance drop at high compression ratios. In this paper, we introduce PuzzleMoE, a training-free MoE compression method that achieves both high accuracy and efficient inference through two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It uses a dual-mask to capture both shared and expert-specific parameters. Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE can compress MoE models by up to 50% while maintaining accuracy across various tasks. Specifically, it outperforms prior MoE compression methods by up to 16.7% on MMLU at 50% compression ratio, and achieves up to 1.28\times inference speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。