提出新方法让模型在不同数据分布下保持稳定梯度方向,提升泛化能力。
Domain Generalization via Pareto Optimal Gradient Matching
- 用元学习框架收集梯度轨迹,独立训练避免方向波动
- 同时最大化跨域梯度内积并限制偏离主优化路径
- 兼顾性能与效率,适合需要鲁棒泛化的实际场景
本文针对基于梯度的域泛化问题,提出一种新的帕累托最优梯度匹配(POGM)方法。现有方法存在两大挑战:一是最小化梯度经验距离或梯度内积(GIP)会导致域间梯度波动,影响学习;二是直接对联合损失进行梯度学习需近似二阶导数,计算开销大。POGM不将梯度匹配作为正则化,而是将梯度轨迹视为训练数据,在元学习器中独立训练。在元更新阶段,最大化各域间GIP,同时限制学习到的梯度偏离经验风险最小化梯度轨迹过远。该策略使聚合梯度能融合所有域的知识,且不偏向任一特定域。在DomainBed数据集上的实验表明,POGM在性能上优于其他基线方法,同时具备更高的计算效率。
原文摘要 · Abstract (English)
In this study, we address the gradient-based domain generalization problem, where predictors aim for consistent gradient directions across different domains. Existing methods have two main challenges. First, minimization of gradient empirical distance or gradient inner products (GIP) leads to gradient fluctuations among domains, thereby hindering straightforward learning. Second, the direct application of gradient learning to the joint loss function can incur high computation overheads due to second-order derivative approximation. To tackle these challenges, we propose a new Pareto Optimality Gradient Matching (POGM) method. In contrast to existing methods that add gradient matching as regularization, we leverage gradient trajectories as collected data and apply independent training at the meta-learner. In the meta-update, we maximize GIP while limiting the learned gradient from deviating too far from the empirical risk minimization gradient trajectory. By doing so, the aggregate gradient can incorporate knowledge from all domains without suffering gradient fluctuation towards any particular domain. Experimental evaluations on datasets from DomainBed demonstrate competitive results yielded by POGM against other baselines while achieving computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。