让神经网络训练时自动聚类权重，提升量化压缩后的精度
Quantization-Aware Regularizers for Deep Neural Networks Compression
- 训练中加入分层正则项，引导权重自然形成可量化聚类
- 在CIFAR-10上用AlexNet和VGG16测试，量化后精度损失显著降低
- 首次将量化中心参数融入反向传播，实现端到端量化感知优化
深度神经网络在众多领域达到顶尖性能，但模型日益庞大且过度参数化，给资源受限设备的部署带来挑战。模型压缩因此变得至关重要，其中权重量化是主流且高效的方法，但通常导致明显的精度下降。现有方法多在模型训练完成后应用，未影响训练过程中的参数空间探索。本文提出每层的正则化项，在训练阶段就驱动权重自然聚类，将量化感知直接融入优化过程。该方法在不牺牲压缩潜力的前提下，有效缓解了量化带来的精度损失。此外，我们的框架首次将量化代表值作为网络参数，实现了量化参数在反向传播中的端到端更新。在CIFAR-10数据集上使用AlexNet和VGG16的实验验证了该策略的有效性。
原文摘要 · Abstract (English)
Deep Neural Networks reached state-of-the-art performance across numerous domains, but this progress has come at the cost of increasingly large and over-parameterized models, posing serious challenges for deployment on resource-constrained devices. As a result, model compression has become essential, and -- among compression techniques -- weight quantization is largely used and particularly effective, yet it typically introduces a non-negligible accuracy drop. However, it is usually applied to already trained models, without influencing how the parameter space is explored during the learning phase. In contrast, we introduce per-layer regularization terms that drive weights to naturally form clusters during training, integrating quantization awareness directly into the optimization process. This reduces the accuracy loss typically associated with quantization methods while preserving their compression potential. Furthermore, in our framework quantization representatives become network parameters, marking, to the best of our knowledge, the first approach to embed quantization parameters directly into the backpropagation procedure. Experiments on CIFAR-10 with AlexNet and VGG16 models confirm the effectiveness of the proposed strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。