GEMQ通过全局优化专家精度分配，提升MoE大模型压缩效率。
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
- 基于量化误差分析的全局线性规划，精准评估专家重要性。
- 引入路由器微调，适应量化后专家分布，减少路由偏差。
- 适用于需要极致压缩的MoE大模型部署场景。
混合专家大语言模型（MoE-LLMs）性能强劲，但因大量专家参数导致显著内存开销。混合精度量化通过根据专家重要性分配不同比特位宽，逼近精度-内存帕累托前沿，实现极低比特量化。然而现有方法依赖层内重要性估计，忽略量化引发的路由器偏移，导致分配不优、路由失准。本文提出全局专家级混合精度量化（GEMQ），通过（1）基于量化误差分析的全局线性规划，捕捉全模型专家重要性；（2）高效路由器微调以适配量化后的专家。二者集成于渐进式量化框架，迭代优化重要性估计与分配。实验表明，GEMQ显著降低内存占用并加速推理，仅带来轻微精度损失。源码见https://github.com/jndeng/GEMQ。
原文摘要 · Abstract (English)
Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation. Source code is available at https://github.com/jndeng/GEMQ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。