通过优化损失曲面一致性,提升模型在未知领域的泛化能力。
Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes
- 迭代反馈训练框架,动态调整不同域的损失曲面
- 在五个基准上平均比尖锐度感知方法高2.6%(ResNet-50)
- 适合需要强跨域泛化的实际应用,如医疗图像分析
领域泛化旨在从多个训练域学习模型,并推广到未见的测试域。近期理论表明,寻找参数位于损失曲面平坦极小值的深度模型,可显著降低域外泛化误差。然而,现有方法常忽略不同域间损失曲面的一致性,导致模型无法同时在所有域中处于最优平坦极小值,限制了泛化性能。为此,本文提出一种迭代自反馈训练(SFT)框架,通过逐步精炼损失曲面,寻求跨域共享的稳定平坦极小值。该框架交替生成反馈信号以衡量各域损失曲面的不一致性,并利用该信号优化曲面一致性。得益于精炼后曲面中平坦极小值的一致性,SFT有效提升了域外泛化能力。在DomainBed上的大量实验表明,相比先进尖锐度感知方法和其他主流领域泛化基线,SFT表现更优:在五种基准上平均分别超越尖锐度最小化方法2.6%(使用ResNet-50)和1.5%(使用ViT-B/16)。代码将尽快公开。
原文摘要 · Abstract (English)
Domain generalization aims to learn a model from multiple training domains and generalize it to unseen test domains. Recent theory has shown that seeking the deep models, whose parameters lie in the flat minima of the loss landscape, can significantly reduce the out-of-domain generalization error. However, existing methods often neglect the consistency of loss landscapes in different domains, resulting in models that are not simultaneously in the optimal flat minima in all domains, which limits their generalization ability. To address this issue, this paper proposes an iterative Self-Feedback Training (SFT) framework to seek consistent flat minima that are shared across different domains by progressively refining loss landscapes during training. It alternatively generates a feedback signal by measuring the inconsistency of loss landscapes in different domains and refines these loss landscapes for greater consistency using this feedback signal. Benefiting from the consistency of the flat minima within these refined loss landscapes, our SFT helps achieve better out-of-domain generalization. Extensive experiments on DomainBed demonstrate superior performances of SFT when compared to state-of-the-art sharpness-aware methods and other prevalent DG baselines. On average across five DG benchmarks, SFT surpasses the sharpness-aware minimization by 2.6% with ResNet-50 and 1.5% with ViT-B/16, respectively. The code will be available soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。