通过动态剪枝与量化压缩专家模型,显著降低大模型存储和推理开销。
MC#: Mixture Compressor for Mixture-of-Experts Large Models
- 结合静态量化与动态剪枝,按专家重要性分配比特数。
- 在五项多模态任务上实现6.2倍权重压缩,仅损失1.7%精度。
- 适合需要高效部署MoE大模型的研究者与工程师。
Mixture-of-Experts(MoE)通过稀疏激活有效扩展大语言模型(LLMs)和视觉语言模型(VLMs)的容量。然而,预加载所有专家并激活多个专家导致显著的计算与内存开销,使专家模块成为模型大小与推理成本的主要来源。为此,我们提出MC#(Mixture-Compressor-sharp),融合静态量化与动态专家剪枝,利用专家和令牌的重要性实现对MoE-LLMs/VLMs的激进压缩。为降低存储与加载成本,引入预加载混合精度量化(PMQ),通过线性规划优化比特分配,在专家重要性与量化误差间实现帕累托最优权衡。为减少运行时计算,提出在线任意剪枝(OTP),使用Gumbel-Softmax采样动态选择每令牌的专家子集,实现细粒度激活控制。结合PMQ的静态比特优化与OTP的动态路由,MC#实现极强压缩且精度损失极小。在DeepSeek-VL2上,MC#实现6.2倍权重压缩,平均比特降至2.57,跨五个多模态基准仅损失1.7%精度。此外,OTP将专家激活率降低超20%,性能下降不足1%,展现出高效部署MoE模型的巨大潜力。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation. However, preloading all experts into memory and activating multiple experts per input introduces significant computational and memory overhead, making the expert module a major contributor to model size and inference cost. To address this, we propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning by leveraging the significance of experts and tokens for aggressive compression of MoE-LLMs/VLMs. To reduce storage and loading costs, we introduce Pre-Loading Mixed-Precision Quantization (PMQ), which optimizes bit allocation via linear programming, balancing expert importance and quantization error for a Pareto-optimal trade-off between size and performance. To reduce runtime computation, Online Top-any Pruning (OTP) uses Gumbel-Softmax sampling to dynamically select a subset of experts per token, enabling fine-grained control over activation. By combining PMQ's static bit-width optimization with OTP's dynamic routing, MC# achieves extreme compression with minimal accuracy loss. On DeepSeek-VL2, MC# achieves a 6.2 times weight reduction at 2.57 average bits with only a 1.7% accuracy drop across five multimodal benchmarks. Additionally, OTP reduces expert activation over 20% with less than 1% performance degradation, demonstrating strong potential for efficient MoE-based model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。