针对低精度模型训练中泛化能力下降问题,提出自适应梯度量化训练方法。
GAQAT: gradient-adaptive quantization-aware training for domain generalization
- 识别低精度量化中的梯度冲突问题,通过调节缩放因子梯度提升训练稳定性。
- 3比特和4比特模型在PACS上性能领先直接融合方法最高4.5%。
- 适用于边缘设备部署,实现接近全精度的域泛化效果,适合资源受限场景。
研究表明,平坦的损失曲面有助于提升模型泛化能力。现有基于平坦性的域泛化(DG)方法多在全精度训练下运行,难以部署于资源受限的边缘设备,而这些设备通常依赖低比特表示(如4比特、3比特)。本文发现,将先进DG-SAM方法应用于量化模型时性能显著下降,表明当前方法无法在低精度训练中保持泛化能力。为此,我们提出一种新的梯度自适应量化感知训练(GAQAT)框架。通过分析发现,量化过程中任务损失与平滑损失对缩放因子产生冲突梯度,导致优化不稳定。为此,我们引入梯度不一致性量化机制,并选择性冻结部分缩放因子梯度以稳定训练过程,从而增强域外泛化能力。大量实验验证了该方法的有效性:在PACS数据集上,3比特和4比特模型相比直接融合方法提升最高达4.5%;在DomainNet上,4比特模型达到近乎无损性能,较最先进量化基线分别提升1.39%(4比特)和1.06%(3比特)。
原文摘要 · Abstract (English)
Research on loss surface geometry, such as Sharpness-Aware Minimization (SAM), shows that flatter minima improve generalization. Recent studies further reveal that flatter minima can also reduce the domain generalization (DG) gap. However, existing flatness-based DG techniques predominantly operate within a full-precision training process, which is impractical for deployment on resource-constrained edge devices that typically rely on lower bit-width representations (e.g., 4 bits, 3 bits). Consequently, low-precision quantization-aware training is critical for optimizing these techniques in real-world applications. In this paper, we observe a significant degradation in performance when applying state-of-the-art DG-SAM methods to quantized models, suggesting that current approaches fail to preserve generalizability during the low-precision training process. To address this limitation, we propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG. Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization, where the task loss and smoothness loss induce conflicting gradients for the scaling factors of quantizers, with certain layers exhibiting opposing gradient directions. This conflict renders the optimization of quantized weights highly unstable. To mitigate this, we further introduce a mechanism to quantify gradient inconsistencies and selectively freeze the gradients of scaling factors, thereby stabilizing the training process and enhancing out-of-domain generalization. Extensive experiments validate the effectiveness of the proposed GAQAT framework. On PACS, our 3-bit and 4-bit models outperform direct DG-QAT integration by up to 4.5%. On DomainNet, the 4-bit model achieves near-lossless performance compared to full precision, with improvements of 1.39% (4-bit) and 1.06% (3-bit) over the SOTA QAT baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。