通过优化生成数据的尖锐度，提升零样本量化模型泛化能力。
Sharpness-Aware Data Generation for Zero-shot Quantization
- 用生成数据与邻近样本的梯度匹配来逼近真实验证集梯度匹配。
- 在CIFAR-100和ImageNet上实现比现有方法更优的低比特量化性能。
- 适合关注模型泛化性与无真实数据量化场景的研究者。
零样本量化旨在不访问原始训练数据的情况下，从预训练的全精度模型中学习量化模型。现有方法通常通过生成合成数据来对全精度模型进行量化。虽然深度神经网络的低尖锐度具有更好泛化能力，但此前的零样本量化方法均未将量化模型的尖锐度作为生成数据的标准。本文提出一种新方法，在合成数据生成中引入量化模型尖锐度的考量，以增强泛化能力。具体而言，我们证明在特定假设下，通过最大化合成数据与真实验证数据在重构损失上的梯度匹配，可实现尖锐度最小化。为解决缺乏真实验证集的问题，我们通过生成样本与其邻近样本间的梯度匹配来近似该目标。在CIFAR-100和ImageNet上的实验表明，该方法在低比特量化设置下显著优于当前最优技术。
原文摘要 · Abstract (English)
Zero-shot quantization aims to learn a quantized model from a pre-trained full-precision model with no access to original real training data. The common idea in zero-shot quantization approaches is to generate synthetic data for quantizing the full-precision model. While it is well-known that deep neural networks with low sharpness have better generalization ability, none of the previous zero-shot quantization works considers the sharpness of the quantized model as a criterion for generating training data. This paper introduces a novel methodology that takes into account quantized model sharpness in synthetic data generation to enhance generalization. Specifically, we first demonstrate that sharpness minimization can be attained by maximizing gradient matching between the reconstruction loss gradients computed on synthetic and real validation data, under certain assumptions. We then circumvent the problem of the gradient matching without real validation set by approximating it with the gradient matching between each generated sample and its neighbors. Experimental evaluations on CIFAR-100 and ImageNet datasets demonstrate the superiority of the proposed method over the state-of-the-art techniques in low-bit quantization settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。