通过量化训练提升跨域泛化能力,无需额外计算开销
QT-DoG: Quantization-aware Training for Domain Generalization
- 用量化引入权重噪声,引导优化走向平坦极小值
- 量化后模型在多个域上准确率超越现有方法
- 可同时压缩模型并提升泛化,适合部署场景
领域泛化(DG)的核心挑战在于防止对源域的过拟合,可通过寻找损失曲面中更平坦的极小值来缓解。本文提出量化感知训练的领域泛化方法(QT-DoG),证明权重量化能有效诱导损失曲面中的平坦极小值,从而增强跨域泛化能力。与侧重模型压缩的传统量化不同,QT-DoG将量化作为隐式正则项,通过在权重中引入噪声,引导优化过程趋向对扰动不敏感、不易过拟合的平坦极小值。我们从理论分析和实证两方面验证了量化天然促进平坦极小值的特性,带来更好的跨域泛化性能。此外,量化带来的模型尺寸缩减使多个量化模型的集成在无额外计算或内存开销下,仍取得优于当前最优DG方法的准确率。代码已开源:https://saqibjaved1.github.io/QT_DoG/
原文摘要 · Abstract (English)
A key challenge in Domain Generalization (DG) is preventing overfitting to source domains, which can be mitigated by finding flatter minima in the loss landscape. In this work, we propose Quantization-aware Training for Domain Generalization (QT-DoG) and demonstrate that weight quantization effectively leads to flatter minima in the loss landscape, thereby enhancing domain generalization. Unlike traditional quantization methods focused on model compression, QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights, guiding the optimization process toward flatter minima that are less sensitive to perturbations and overfitting. We provide both an analytical perspective and empirical evidence demonstrating that quantization inherently encourages flatter minima, leading to better generalization across domains. Moreover, with the benefit of reducing the model size through quantization, we demonstrate that an ensemble of multiple quantized models further yields superior accuracy than the state-of-the-art DG approaches with no computational or memory overheads. Code is released at: https://saqibjaved1.github.io/QT_DoG/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。