arXiv:2503.04315cs.LGcs.AI2025-03ICLR

新方法缓解了Wasserstein鲁棒优化的过拟合问题,提升模型在对抗样本下的泛化能力。

Provable Robust Overfitting Mitigation in Wasserstein Distributionally Robust Optimization

论文配图:Provable Robust Overfitting Mitigation in Wasserstein Distributionally Robust Optimization
图 1 · 摘自论文原文
  • 结合Wasserstein距离与KL散度构建新型不确定集,同时考虑分布扰动和统计误差。
  • 理论证明:模型在分布外对抗样本上的表现至少等于统计鲁棒训练损失。
  • 实验验证该方法显著降低鲁棒过拟合,适用于高可靠性要求的对抗训练场景。

Wasserstein分布鲁棒优化(WDRO)通过在指定不确定性集中优化最坏情况下的分布偏移,相比仅关注点级对抗扰动的标准对抗训练,能更好提升对未见对抗样本的泛化能力。然而,WDRO仍存在根本性问题:未考虑统计误差,导致鲁棒过拟合。本文提出一种新框架——统计鲁棒WDRO,采用Wasserstein距离刻画对抗噪声的不确定性集,同时以KL散度衡量统计误差。我们建立了该框架的鲁棒泛化界,表明在高概率下,模型在分布外对抗样本上的性能至少不劣于统计鲁棒训练损失。进一步推导出学习者与对手间存在斯塔克尔伯格与纳什均衡的条件,从而获得某种意义上的最优鲁棒模型。大量实验表明,该方法能显著缓解鲁棒过拟合,并在WDRO框架内增强模型鲁棒性。

原文摘要 · Abstract (English)

Wasserstein distributionally robust optimization (WDRO) optimizes against worst-case distributional shifts within a specified uncertainty set, leading to enhanced generalization on unseen adversarial examples, compared to standard adversarial training which focuses on pointwise adversarial perturbations. However, WDRO still suffers fundamentally from the robust overfitting problem, as it does not consider statistical error. We address this gap by proposing a novel robust optimization framework under a new uncertainty set for adversarial noise via Wasserstein distance and statistical error via Kullback-Leibler divergence, called the Statistically Robust WDRO. We establish a robust generalization bound for the new optimization framework, implying that out-of-distribution adversarial performance is at least as good as the statistically robust training loss with high probability. Furthermore, we derive conditions under which Stackelberg and Nash equilibria exist between the learner and the adversary, giving an optimal robust model in certain sense. Finally, through extensive experiments, we demonstrate that our method significantly mitigates robust overfitting and enhances robustness within the framework of WDRO.

鲁棒优化分布鲁棒对抗训练泛化界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。