arXiv:2606.04980cs.LG2026-06

无需校准数据,根据专家权重分布特性自动分配比特数,显著压缩MoE模型内存占用。

AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

论文配图:AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
图 1 · 摘自论文原文
  • 基于权重谱的重尾特性判断专家重要性,实现无校准比特分配
  • 在相同比特预算下,比依赖校准的方法更优,Qwen1.5-MoE仅用3.5比特平均精度接近全精度
  • 特别适合无法获取训练数据的前沿MoE大模型部署场景

Mixture-of-Experts (MoE) 架构通过稀疏专家激活扩展模型容量,但部署时受内存限制,因所有专家权重需驻留内存。混合精度量化可通过为不同专家分配不同比特数显著降低内存占用。然而,现有方法通常依赖校准数据估计专家重要性并确定比特分配。对于前沿 MoE 大模型,原始训练数据(即真实训练分布)为专有且不可访问,导致校准集成为不可避免的近似,可能误估专家使用率,进而导致次优比特分配。受现代 MoE 模型中跨专家质量差异显著及重尾自正则化(HT-SR)理论在无训练或测试数据情况下预测模型质量的成功启发,我们提出 AlphaQ:一种面向 MoE 量化的无校准比特分配方法。AlphaQ 基于 HT-SR 理论,遵循简单原则:具有更强重尾权重谱的专家通常训练更充分,应分配更高比特数;而重尾结构较弱的专家可更激进地量化。AlphaQ 通过测量专家级谱重尾性,并求解在全局比特预算约束下的最小总量化误差优化问题来实现该原则。在多个 MoE 模型上,AlphaQ 在匹配比特预算下持续优于基于校准的基线方法。值得注意的是,在 Qwen1.5-MoE 上,AlphaQ 以仅 3.5 比特的平均专家精度实现了接近全精度的准确率,同时带来超过 4× 的内存压缩。代码已开源:https://github.com/Superone77/AlphaQ。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in memory. Mixed-precision quantization can substantially reduce this footprint by assigning different bit-widths to different experts. Existing approaches, however, typically rely on calibration data to estimate expert importance and determine bit allocation. For frontier MoE LLMs, the original training data, and hence the true training distribution, is proprietary and inaccessible. As a result, calibration sets are inevitably imperfect surrogates, and this can misestimate expert utilization and lead to suboptimal bit allocation. Motivated by the substantial cross-expert quality variability observed in modern MoE models, and by the success of Heavy-Tailed Self-Regularization (HT-SR) theory at predicting neural network model quality without access to training or testing data, we propose AlphaQ, a calibration-free bit-allocation method for MoE quantization. AlphaQ draws on HT-SR theory and follows a simple principle: experts with more heavy-tailed weight spectra are typically better trained and hence should receive higher bit-widths, while experts with weaker heavy-tailed structure can be quantized more aggressively. AlphaQ operationalizes this principle by measuring expert-wise spectral heavy-tailedness and solving a budget-constrained optimization problem that minimizes total quantization error under a global bit-budget constraint. Across several MoE models, AlphaQ consistently outperforms calibration-based baselines under matched bit budgets. Notably, on Qwen1.5-MoE, AlphaQ achieves near full-precision accuracy with an average expert precision of only 3.5 bits, while delivering more than 4$\times$ memory compression. Our code is available at https://github.com/Superone77/AlphaQ.

MoE量化无校准比特分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。