通过特征扰动稳定量化训练，提升模型压缩效果
Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix
- 在特征层随机加扰动，缓解量化导致的损失曲面陡峭
- 实验显示方法优于现有最先进量化训练方案
- 适用于多种网络结构和量化方法，特别适合压缩场景
量化感知训练（QAT）是主流的神经网络压缩方法，但其稳定性因不可避免的量化误差而受到挑战。我们发现，损失函数的陡峭地形是导致性能下降的关键原因。理论上，特征扰动可使局部极小值变平坦。然而，直接对权重或特征添加扰动会损害全精度（FP）模型性能。本文提出特征扰动量化（FPQ），在特征层施加随机扰动，并采用特征蒸馏方法优化量化模型。数学上证明，该方法隐式正则化了海森矩阵范数，从而平滑损失曲面。大量实验表明，本方法显著优于当前SOTA QAT方法，甚至超越部分全精度模型。
原文摘要 · Abstract (English)
Quantization-Aware Training (QAT) is one of the prevailing neural network compression solutions. However, its stability has been challenged for yielding deteriorating performances as the quantization error is inevitable. We find that the sharp landscape of loss, which leads to a dramatic performance drop, is an essential factor that causes instability. Theoretically, we have discovered that the perturbations in the feature would bring a flat local minima. However, simply adding perturbations into either weight or feature empirically deteriorates the performance of the Full Precision (FP) model. In this paper, we propose Feature-Perturbed Quantization (FPQ) to stochastically perturb the feature and employ the feature distillation method to the quantized model. Our method generalizes well to different network architectures and various QAT methods. Furthermore, we mathematically show that FPQ implicitly regularizes the Hessian norm, which calibrates the smoothness of a loss landscape. Extensive experiments demonstrate that our approach significantly outperforms the current State-Of-The-Art (SOTA) QAT methods and even the FP counterparts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。