聚焦归一化参数优化,显著提升低比特量化精度
Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization

- 识别归一化仿射参数为关键低维子空间,针对性优化提升抗量化能力
- 在ImageNet和CIFAR-100上,低比特量化准确率大幅恢复,超越全参数联合训练
- 适用于轻量模型量化,对预训练与微调均有效,计算开销极低
低比特量化在紧凑网络中面临严重精度下降,根源在于主流的全参数联合训练范式忽略了参数子空间异质性。其有限的特征冗余使量化误差难以吸收。传统方法采用单一优化策略:PTQ在固定预训练模型上重建,无法改善固有量化友好性;QAT联合更新所有参数,受主干权重与校准参数间梯度耦合影响。本文识别出归一化仿射参数构成主导量化鲁棒性的低维高杠杆子空间,提出归一化仿射预处理(NAP)进行定向子空间优化。对于PTQ,NAP冻结主干权重,在全精度模型上仅微调仿射参数,于目标伪量化图下主动增强量化友好性;对于QAT,引入交替式QAT-NAP框架,解耦特征学习与数值校准,突破饱和联合训练性能上限。理论分析表明,BN仿射参数可完全抵消量化失真中的通道级仿射成分,而非线性舍入与截断残差构成不可消除误差边界;基于知识蒸馏的NAP实现方向性平坦度优化,将教师-学生逻辑差异投影至受限子空间。ImageNet与CIFAR-100实验显示,NAP能有效恢复严重退化的低比特量化性能,持续提升基于重建的PTQ表现,并以极小调参成本优于饱和全参数QAT。本工作揭示了定向低维子空间优化的原则,为高效深度学习提供了超越全参数联合训练的新视角。
原文摘要 · Abstract (English)
Low-bit quantization suffers severe accuracy degradation on compact networks, rooted in the dominant full-parameter coupled training paradigm that ignores parameter subspace heterogeneity. Their limited feature redundancy leaves little room to absorb quantization errors. Conventional pipelines adopt monolithic optimization: PTQ reconstructs fixed pretrained models without improving inherent quantization friendliness; QAT updates all parameters jointly, suffering from gradient coupling between backbone weights and calibration parameters. In this paper, we identify normalization affine parameters as a low-dimensional high-leverage subspace dominating quantization robustness, and propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization. For PTQ, NAP freezes backbone weights and fine-tunes only affine parameters under the target fake-quantization graph on full-precision models, proactively boosting quantization friendliness before downstream reconstruction. For QAT, we introduce an alternating QAT-NAP schema that decouples feature learning and numerical calibration, breaking the performance ceiling of saturated joint training. Theoretical analysis confirms BN affine parameters fully cancel the channel-wise affine component of quantization distortion, while nonlinear rounding and clipping residuals form the irreducible error boundary; distillation-guided NAP acts as directional flatness optimization, projecting teacher-student logit mismatch onto the restricted subspace. Experiments on ImageNet and CIFAR-100 show NAP recovers severely collapsed low-bit quantization, consistently boosts reconstruction-based PTQ, and outperforms saturated full-parameter QAT with negligible tuning cost. This work reveals the principle of targeted low-dimensional subspace optimization, offering a new perspective beyond full-parameter coupled training for efficient deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。