arXiv:2412.15846cs.LG2024-12被引 10

通过替换全精度模型块,提升低精度网络量化训练效果。

Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart

  • 用全精度模型块指导低精度网络的前向与反向传播
  • 4/3/2比特量化在ImageNet和CIFAR-10上达领先性能
  • 兼容主流量化训练方法,代码改造极简

量化感知训练(QAT)通过模拟低精度计算来优化量化参数。但直接训练低精度网络存在两大问题:一是表达能力弱,难以复现全精度计算;二是伪梯度近似导致梯度传播偏差。本文提出一种通用框架,让低精度网络的前向与反向过程由其全精度对应模型引导。通过逐块替换生成中间混合精度模型,使量化块在训练中同时存在于全精度网络中。这样每个量化块可实现:1. 前向传递时模拟全精度表示;2. 反向传播时获得更优梯度估计。实验表明,该方法在ImageNet和CIFAR-10上对4、3、2比特量化均达到当前最优结果。框架可兼容多数现有QAT方法,仅需简洁封装即可集成。

原文摘要 · Abstract (English)

Quantization-aware training (QAT) is a common paradigm for network quantization, in which the training phase incorporates the simulation of the low-precision computation to optimize the quantization parameters in alignment with the task goals. However, direct training of low-precision networks generally faces two obstacles: 1. The low-precision model exhibits limited representation capabilities and cannot directly replicate full-precision calculations, which constitutes a deficiency compared to full-precision alternatives; 2. Non-ideal deviations during gradient propagation are a common consequence of employing pseudo-gradients as approximations in derived quantized functions. In this paper, we propose a general QAT framework for alleviating the aforementioned concerns by permitting the forward and backward processes of the low-precision network to be guided by the full-precision partner during training. In conjunction with the direct training of the quantization model, intermediate mixed-precision models are generated through the block-by-block replacement on the full-precision model and working simultaneously with the low-precision backbone, which enables the integration of quantized low-precision blocks into full-precision networks throughout the training phase. Consequently, each quantized block is capable of: 1. simulating full-precision representation during forward passes; 2. obtaining gradients with improved estimation during backward passes. We demonstrate that the proposed method achieves state-of-the-art results for 4-, 3-, and 2-bit quantization on ImageNet and CIFAR-10. The proposed framework provides a compatible extension for most QAT methods and only requires a concise wrapper for existing codes.

量化训练混合精度神经网络压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。