提出新方法提升低精度神经网络训练效果,让模型更准更快。
Progressive Element-wise Gradient Estimation for Neural Network Quantization
- 用渐进式逐元素梯度估计替代传统方法
- 在极低比特下仍保持高精度,部分超过全精度模型
- 适合部署到资源受限设备的低精度模型训练
神经网络量化通过降低权重和激活的位宽,使深度神经网络能在资源受限硬件上高效部署。现有量化感知训练(QAT)方法多采用直通估计器(STE)处理离散化函数不可导问题,用恒等函数导数替代,但忽略了连续与量化值间的误差,导致低比特下精度下降。本文提出渐进式逐元素梯度估计(PEGE),可无缝集成于任意前向传播方式。PEGE通过新型对数课程驱动的混合精度替换策略,逐步将全精度参数替换为量化值,并将QAT建模为联合优化任务:同时最小化预测损失与量化误差,形成统一通用框架。在CIFAR-10和ImageNet上对ResNet、VGG等多种架构的实验表明,PEGE持续优于现有反向传播方法,使低精度模型达到甚至超越全精度模型的准确率。
原文摘要 · Abstract (English)
Neural network quantization aims to reduce the bit-widths of weights and activations, making it a critical technique for deploying deep neural networks on resource-constrained hardware. Most Quantization-Aware Training (QAT) methods rely on the Straight-Through Estimator (STE) to address the non-differentiability of discretization functions by replacing their derivatives with that of the identity function. While effective, STE overlooks discretization errors between continuous and quantized values, which can lead to accuracy degradation -- especially at extremely low bit-widths. In this paper, we propose Progressive Element-wise Gradient Estimation (PEGE), a simple yet effective alternative to STE, which can be seamlessly integrated with any forward propagation methods and improves the quantized model accuracy. PEGE progressively replaces full-precision weights and activations with their quantized counterparts via a novel logarithmic curriculum-driven mixed-precision replacement strategy. Then it formulates QAT as a co-optimization problem that simultaneously minimizes the task loss for prediction and the discretization error for quantization, providing a unified and generalizable framework. Extensive experiments on CIFAR-10 and ImageNet across various architectures (e.g., ResNet, VGG) demonstrate that PEGE consistently outperforms existing backpropagation methods and enables low-precision models to match or even outperform the accuracy of their full-precision counterparts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。