通过梯度外推让模型忽略无关特征,提升泛化能力。
Gradient Extrapolation for Debiased Representation Learning
- 用两组不同干扰特征的数据计算梯度,线性外推得到更优方向。
- 在5个视觉和1个NLP任务上,优于或媲美当前最佳方法。
- 可调节参数适配不同评估指标,通用性强,适合多场景应用。
基于经验风险最小化的分类模型常依赖虚假相关性,导致测试时泛化性能下降。本文从优化视角提出梯度外推去偏表示学习(GERNE)方法,在已知和未知属性训练情形下均能学习去偏表示。GERNE利用两组具有不同虚假相关性强度的批次数据,将目标梯度定义为各批次损失梯度的线性外推。理论分析表明,当外推梯度指向虚假相关性较少的批次梯度方向时,可有效引导模型学习去偏表示。GERNE是一个通用去偏框架,包含ERM与重采样方法作为特例。本文推导了外推因子的理论上下界,通过调节该因子可最大化群体平衡准确率(GBA)或最差群体准确率(WGA)。在五个视觉和一个NLP基准上验证,GERNE表现竞争力甚至超越现有最优方法。
原文摘要 · Abstract (English)
Machine learning classification models trained with empirical risk minimization (ERM) often inadvertently rely on spurious correlations. When absent in the test data, these unintended associations between non-target attributes and target labels lead to poor generalization. This paper addresses this problem from a model optimization perspective and proposes a novel method, Gradient Extrapolation for Debiased Representation Learning (GERNE), designed to learn debiased representations in both known and unknown attribute training cases. GERNE uses two distinct batches with different amounts of spurious correlations and defines the target gradient as a linear extrapolation of the gradients computed from each batch's loss. Our analysis shows that when the extrapolated gradient points toward the batch gradient with fewer spurious correlations, it effectively guides training toward learning a debiased model. GERNE serves as a general framework for debiasing, encompassing ERM and Resampling methods as special cases. We derive the theoretical upper and lower bounds of the extrapolation factor employed by GERNE. By tuning this factor, GERNE can adapt to maximize either Group-Balanced Accuracy (GBA) or Worst-Group Accuracy (WGA). We validate GERNE on five vision and one NLP benchmarks, demonstrating competitive and often superior performance compared to state-of-the-art baselines. The project page is available at: https://gerne-debias.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。