arXiv:2511.05568cs.LGcs.CV2025-11

为提升模型在分布偏移下的泛化能力,提出自适应样本级鲁棒优化方法

Adaptive Sample-Level Framework Motivated by Distributionally Robust Optimization with Variance-Based Radius Assignment for Enhanced Neural Network Generalization Under Distribution Shift

  • 根据样本损失方差动态分配个性化鲁棒预算,实现精细化风险控制
  • 在CIFAR-10-C和Waterbirds上均优于ERM和KL-DRO,整体准确率最高
  • 无需分组标签、易实现,适合关注模型鲁棒性的研究者使用

分布偏移和少数子群体常导致基于经验风险最小化的深度神经网络可靠性下降。分布鲁棒优化(DRO)通过优化训练分布邻域内的最差情况风险来应对这一问题。然而,传统方法依赖单一全局鲁棒预算,可能导致模型过度保守或鲁棒性分配不当。本文提出一种基于方差驱动的自适应样本级DRO(Var-DRO)框架,自动识别高风险训练样本,并根据其在线损失方差为每个样本分配个性化鲁棒预算。该方法采用双侧KL散度风格的约束,限制每个样本的对抗权重与经验权重之比,形成凸多面体上的线性内最大化问题,可高效求解。为稳定训练,引入预热阶段和线性增长的全局预算上限,并结合标签平滑提升数值鲁棒性。在CIFAR-10-C(含扰动数据集)上,本方法取得最高平均准确率;在Waterbirds上性能优于或媲美KL-DRO;在原始CIFAR-10上保持竞争力,仅小幅牺牲精度以换取更强鲁棒性。该框架无需分组标签,实现简单,理论严谨且计算高效。

原文摘要 · Abstract (English)

Distribution shifts and minority subpopulations frequently undermine the reliability of deep neural networks trained using Empirical Risk Minimization (ERM). Distributionally Robust Optimization (DRO) addresses this by optimizing for the worst-case risk within a neighborhood of the training distribution. However, conventional methods depend on a single, global robustness budget, which can lead to overly conservative models or a misallocation of robustness. We propose a variance-driven, adaptive, sample-level DRO (Var-DRO) framework that automatically identifies high-risk training samples and assigns a personalized robustness budget to each based on its online loss variance. Our formulation employs two-sided, KL-divergence-style bounds to constrain the ratio between adversarial and empirical weights for every sample. This results in a linear inner maximization problem over a convex polytope, which admits an efficient water-filling solution. To stabilize training, we introduce a warmup phase and a linear ramp schedule for the global cap on per-sample budgets, complemented by label smoothing for numerical robustness. Evaluated on CIFAR-10-C (corruptions), our method achieves the highest overall mean accuracy compared to ERM and KL-DRO. On Waterbirds, Var-DRO improves overall performance while matching or surpassing KL-DRO. On the original CIFAR-10 dataset, Var-DRO remains competitive, exhibiting the modest trade-off anticipated when prioritizing robustness. The proposed framework is unsupervised (requiring no group labels), straightforward to implement, theoretically sound, and computationally efficient.

分布鲁棒自适应优化泛化能力样本级

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。