让神经网络权重振荡，能显著提升量化后的鲁棒性
Oscillations Make Neural Networks Robust to Quantization
- 通过分析发现量化训练中的权重振荡是提升性能的关键机制
- 在多种量化级别下，振荡训练后直接量化可接近全量化训练效果
- 适合关注模型压缩与量化训练的工程师和研究者
我们挑战了当前认为量化感知训练（QAT）中权重振荡仅为副作用的观点，提出振荡实为QAT的重要组成部分。在单变量线性模型中，我们发现QAT引入额外损失项，使权重远离最近的量化等级，从而产生振荡。基于此机制，我们设计了一种正则化器，在训练中主动诱导权重振荡。在ResNet-18和Tiny Vision Transformer上，于CIFAR-10和Tiny ImageNet数据集上的实验表明，采用振荡训练后进行后训练量化（PTQ），在多种量化精度下几乎可完全恢复QAT性能。本工作深化了对QAT动态过程的理解，揭示了振荡在量化中的积极作用，此前其多被视为负面影响。
原文摘要 · Abstract (English)
We challenge the prevailing view that weight oscillations observed during Quantization Aware Training (QAT) are merely undesirable side-effects and argue instead that they are an essential part of QAT. We show in a univariate linear model that QAT results in an additional loss term that causes oscillations by pushing weights away from their nearest quantization level. Based on the mechanism from the analysis, we then derive a regularizer that induces oscillations in the weights of neural networks during training. Our empirical results on ResNet-18 and Tiny Vision Transformer, evaluated on CIFAR-10 and Tiny ImageNet datasets, demonstrate across a range of quantization levels that training with oscillations followed by post-training quantization (PTQ) is sufficient to recover the performance of QAT in most cases. With this work we provide further insight into the dynamics of QAT and contribute a novel insight into explaining the role of oscillations in QAT which until now have been considered to have a primarily negative effect on quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。