改进分布外泛化能力，让模型在未知数据上更稳定
Out-of-distribution Generalization for Total Variation based Invariant Risk Minimization
- 将IRMTV转化为拉格朗日对偶优化框架，自动调节正则强度
- 在多个数据集上显著提升分布外性能，优于传统IRM-TV
- 适合追求鲁棒性、需处理分布偏移的机器学习研究者
不变风险最小化（IRM）是一种重要的通用机器学习框架，近期被解释为全变差模型（IRM-TV）。然而，如何提升在IRM-TV设置下的分布外（OOD）泛化能力仍未解决。本文将IRM-TV扩展为名为OOD-TV-IRM的拉格朗日乘子模型。我们发现，自主的全变差惩罚超参数恰好是拉格朗日乘子，因此OOD-TV-IRM本质上是一个原-对偶优化模型：原问题最小化整体不变风险，对偶问题强化全变差惩罚。目标是达到半纳什平衡，维持训练损失与分布外泛化之间的平衡。我们还设计了一种收敛的原-对偶算法，支持对抗学习机制。实验结果表明，OOD-TV-IRM在多数情况下优于IRM-TV。
原文摘要 · Abstract (English)
Invariant risk minimization is an important general machine learning framework that has recently been interpreted as a total variation model (IRM-TV). However, how to improve out-of-distribution (OOD) generalization in the IRM-TV setting remains unsolved. In this paper, we extend IRM-TV to a Lagrangian multiplier model named OOD-TV-IRM. We find that the autonomous TV penalty hyperparameter is exactly the Lagrangian multiplier. Thus OOD-TV-IRM is essentially a primal-dual optimization model, where the primal optimization minimizes the entire invariant risk and the dual optimization strengthens the TV penalty. The objective is to reach a semi-Nash equilibrium where the balance between the training loss and OOD generalization is maintained. We also develop a convergent primal-dual algorithm that facilitates an adversarial learning scheme. Experimental results show that OOD-TV-IRM outperforms IRM-TV in most situations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。