BitsMoE通过频谱能量引导,实现MoE大模型超低比特量化,显著提升效率与精度。
BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization
- 基于SVD分解专家层,用共享基和特定频谱因子作为量化单元。
- 在2比特下比GPTQ平均准确率高27.83个百分点,推理速度提升1.76倍。
- 适合部署资源受限的MoE大模型,尤其适用于超低比特场景。
混合专家(MoE)大语言模型通过稀疏专家激活降低每令牌计算量,但部署时仍需将所有专家权重驻留内存,导致内存占用高。现有压缩方法在超低比特环境下表现不佳:剪枝会不可逆地损失模型容量,粗粒度量化无法根据专家和权重方向的重要性分配比特。本文提出BitsMoE,一种面向MoE大模型量化的频谱能量引导比特分配框架。通过SVD将每个MoE层分解为共享基和专家特异性频谱因子,保留共享基不量化以维持跨专家共性结构,并以专家特异性因子作为细粒度量化单元。为确定各单元比特宽度,将频谱级混合精度量化建模为感知激活的重构代理,求解在固定比特预算下的整数线性规划以最小化估计重构误差。在多个MoE大模型上的实验表明,BitsMoE在超低比特条件下显著减少下游任务精度下降。在Qwen3-30B-A3B-Base模型上,2比特量化下,相比GPTQ,量化加速12.3倍,平均准确率提升27.83个百分点,解码速度提高1.76倍。代码与模型已公开于https://github.com/zjiayu064/BitsMoE。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation, but their deployment remains memory-intensive because all expert weights must be kept resident in memory. Existing MoE compression methods struggle in the ultra-low-bit regime: pruning irreversibly removes model capacity, while coarse-grained quantization fails to allocate bits according to heterogeneous expert and weight-direction importance. We propose BitsMoE, a spectral-energy-guided bit-allocation framework for MoE LLM quantization. BitsMoE decomposes each MoE layer by SVD into a shared basis and expert-specific spectral factors, retaining the shared basis without quantization to preserve common cross-expert structure and using the expert-specific factors as fine-grained quantization units. To determine the bit-width of each unit, BitsMoE formulates spectrum-wise mixed-precision quantization as an activation-aware reconstruction surrogate and solves an integer linear program that minimizes estimated reconstruction loss under a fixed bit budget. Experiments across multiple MoE LLMs show that BitsMoE substantially reduces downstream task accuracy degradation in ultra-low-bit regimes. Under 2-bit quantization on Qwen3-30B-A3B-Base, BitsMoE accelerates quantization by 12.3$\times$, improves average accuracy by 27.83 percentage points, and increases decoding speed by 1.76$\times$ over GPTQ. Our model and code are publicly available at https://github.com/zjiayu064/BitsMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。