针对神经网络量化中的分布不均和码本固定问题,提出自适应混合精度量化方法。
Adaptive Distribution-aware Quantization for Mixed-Precision Neural Networks
- 根据权重分布分位数初始化码本,动态用EMA更新以追踪变化。
- 在ImageNet上用2.81比特平均位宽实现71.512%准确率,优于现有方法。
- 适合资源受限设备部署,尤其对非均匀分布权重敏感的场景。
量化感知训练(QAT)是将深度神经网络部署到资源受限设备的关键技术。然而,现有方法常面临两大挑战:激活值分布高度非均匀,以及权重量化中使用静态且不匹配的码本。为此,我们提出自适应分布感知量化(ADQ),一种混合精度量化框架,采用差异化策略。ADQ的核心是一种新颖的自适应权重量化方案,包含三项创新:(1)基于分位数的初始化方法,构建与初始权重分布紧密对齐的码本;(2)基于指数移动平均(EMA)的在线码本自适应机制,动态跟踪分布偏移;(3)基于敏感度的混合精度分配策略。对于激活值,集成硬件友好的非均匀到均匀映射方案。全面实验验证了该方法的有效性。在ImageNet上,ADQ使ResNet-18达到71.512%的Top-1准确率,平均位宽仅2.81比特,优于同类先进方法。此外,在CIFAR-10上的详尽消融实验系统验证了各组件的独立贡献,支持设计合理性与有效性。
原文摘要 · Abstract (English)
Quantization-Aware Training (QAT) is a critical technique for deploying deep neural networks on resource-constrained devices. However, existing methods often face two major challenges: the highly non-uniform distribution of activations and the static, mismatched codebooks used in weight quantization. To address these challenges, we propose Adaptive Distribution-aware Quantization (ADQ), a mixed-precision quantization framework that employs a differentiated strategy. The core of ADQ is a novel adaptive weight quantization scheme comprising three key innovations: (1) a quantile-based initialization method that constructs a codebook closely aligned with the initial weight distribution; (2) an online codebook adaptation mechanism based on Exponential Moving Average (EMA) to dynamically track distributional shifts; and (3) a sensitivity-informed strategy for mixed-precision allocation. For activations, we integrate a hardware-friendly non-uniform-to-uniform mapping scheme. Comprehensive experiments validate the effectiveness of our method. On ImageNet, ADQ enables a ResNet-18 to achieve 71.512% Top-1 accuracy with an average bit-width of only 2.81 bits, outperforming state-of-the-art methods under comparable conditions. Furthermore, detailed ablation studies on CIFAR-10 systematically demonstrate the individual contributions of each innovative component, validating the rationale and effectiveness of our design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。