通过梯度引导的参数退火,提升模型对未知数据分布的泛化能力。
Gradient-Guided Annealing for Domain Generalization
- 训练初期动态调整参数,使不同领域梯度方向对齐。
- 在五个图像分类基准上达到顶尖或接近顶尖性能。
- 可与现有方法结合,显著提升其泛化效果。
领域泛化(DG)研究近年来备受关注,因即使最先进的训练算法也难以在未见数据分布上保持良好表现。本文发现,模型训练的初始阶段对领域泛化效果至关重要,因为训练与测试分布的损失景观可能存在显著差异,不同于独立同分布(i.i.d.)数据的情形。各领域损失分量之间的梯度冲突会引导优化过程进入无法捕捉目标类别领域不变特征的不良局部极小值。为此,我们提出梯度引导退火(GGA)算法,在训练早期迭代地退火模型参数,寻找使各领域梯度方向一致的参数点。通过找到一组参数值,使所有训练数据分布的梯度更新方向趋同,该方法促使模型探索更具鲁棒性的最小值。GGA 在五个广泛认可且具有挑战性的图像分类领域泛化基准上进行了评估,仅使用 GGA 即可实现高度竞争性甚至领先于当前最优的结果。此外,将其与已有领域泛化算法结合,能持续显著提升其性能。
原文摘要 · Abstract (English)
Domain Generalization (DG) research has gained considerable traction as of late, since the ability to generalize to unseen data distributions is a requirement that eludes even state-of-the-art training algorithms. In this paper we observe that the initial iterations of model training play a key role in domain generalization effectiveness, since the loss landscape may be significantly different across the training and test distributions, contrary to the case of i.i.d. data. Conflicts between gradients of the loss components of each domain lead the optimization procedure to undesirable local minima that do not capture the domain-invariant features of the target classes. We propose alleviating domain conflicts in model optimization, by iteratively annealing the parameters of a model in the early stages of training and searching for points where gradients align between domains. By discovering a set of parameter values where gradients are updated towards the same direction for each data distribution present in the training set, the proposed Gradient-Guided Annealing (GGA) algorithm encourages models to seek out minima that exhibit improved robustness against domain shifts. The efficacy of GGA is evaluated on five widely accepted and challenging image classification domain generalization benchmarks, where its use alone is able to establish highly competitive or even state-of-the-art performance. Moreover, when combined with previously proposed domain-generalization algorithms it is able to consistently improve their effectiveness by significant margins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。