针对稀疏专家模型量化难题,提出专家感知优化框架,显著提升极低比特下的精度与稳定性。
EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
- 通过专家激活平滑、路由一致性对齐和校准数据均衡三策略,增强量化鲁棒性。
- 在W4A4至W2A4等极端量化条件下,平均准确率提升1.15%~13.81%。
- 特别适合需要高效压缩的大规模推理模型,尤其在复杂任务中表现突出。
混合专家(MoE)模型在大规模深度学习中实现可扩展计算与性能,但因专家稀疏激活与动态路由面临量化挑战。现有后训练量化(PTQ)方法无法应对激活异常值、路由不稳与稀疏专家校准问题,导致性能严重下降。为此,我们提出专为MoE架构设计的EAQuant框架,引入三项专家感知创新:(1) 激活平滑聚合以抑制异常值,(2) 路由一致性对齐以保持量化后专家选择,(3) 校准数据平衡以优化稀疏激活专家。这些策略协同作用,在超低比特约束下实现鲁棒高精度量化。在多个极端量化设置(如W4A4/W3A4/W3A3/W2A4)下广泛实验表明,EAQuant显著优于现有方法,跨三种不同MoE架构平均准确率提升1.15%–13.81%,在推理任务中增益尤为明显,且在激进量化下仍保持强性能。通过整合上述创新,EAQuant确立了高精度、高效MoE模型压缩的新基准。代码已开源。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models enable scalable computation and performance in large-scale deep learning but face quantization challenges due to sparse expert activation and dynamic routing. Existing post-training quantization (PTQ) methods fail to address activation outliers, routing instability, and sparse expert calibration, leading to significant performance degradation. To address this, we propose EAQuant, a PTQ framework tailored for MoE architectures. Our method introduces three expert-aware innovations: (1) smoothing aggregation to suppress activation outliers, (2) routing consistency alignment to preserve expert selection post-quantization, and (3) calibration data balance to optimize sparsely activated experts. These strategies collectively enable robust, high-precision quantization of MoE models under ultra-low-bit constraints.Extensive experiments across several extreme quantization settings (e.g., W4A4/W3A4/W3A3/W2A4) demonstrate that EAQuant significantly outperforms existing methods, achieving average accuracy improvements of 1.15 - 13.81% across three diverse MoE architectures, with particularly pronounced gains in reasoning tasks and robust performance retention under aggressive quantization. By integrating these innovations, EAQuant establishes a new state-of-the-art for high-precision, efficient MoE model compression.Our code is available at https://github.com/darren-fzq1/EAQuant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。