通过反事实增强训练,让模型不再依赖局部线索,提升鲁棒性。
Did Models Learn Sufficiently? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation
- 基于归因分析筛选关键区域,构造反事实样本进行训练
- 在ImageNet系列上实现最高18.04%的外部泛化性能提升
- 适合关注模型可靠性与泛化能力的研究者
当前视觉模型常依赖有限的判别性视觉线索进行预测,在分布偏移或线索缺失时表现不可靠。忠实归因方法可揭示此类问题,但通常仅用于事后分析,未反馈至模型训练。为此,本文提出子集选择式反事实增强(SS-CA),通过遮蔽决策相关区域生成反事实样本,并引导模型建立更鲁棒的决策边界。具体地,将基于子集选择的忠实归因方法LIMA扩展为反事实LIMA,识别移除后使模型转向竞争类别的区域;随后选取近边界的遮蔽方案,在保持原语义的前提下缩小对数概率差,并采用自适应填充策略替换被遮区域,避免引入外部语义。将这些反事实样本回传训练,促使模型利用剩余有效信息,推动决策边界向更鲁棒方向迁移。在五个ImageNet变体上的实验表明,SS-CA显著提升内部准确率、外部泛化能力及扰动鲁棒性:在CLIP ViT/32b下,ImageNet-1k和ImageNet-R分别提升5.70%和18.04%;在ResNet-101与TinyImageNet-200上,ImageNet-R和ImageNet-S分别提升9.52%和11.33%;高斯噪声扰动下约提升4%。代码即将发布。
原文摘要 · Abstract (English)
Current visual models often make predictions based on a limited set of discriminative visual cues. As a result, they may become unreliable when the distribution shifts or when these cues are missing. Faithful attribution methods can reveal such problematic reliance through localized explanations, but they are typically used post hoc and are not fed back into the model. To address this limitation, we propose Subset-Selected Counterfactual Augmentation (SS-CA), a training strategy that masks decision-relevant regions to construct counterfactual samples and guide the model toward more robust decision boundaries. Specifically, we extend LIMA, a subset-selection-based faithful attribution method, to Counterfactual LIMA to identify regions whose removal shifts the model toward a competing class. SS-CA then selects near-boundary masks that reduce the logit gap while preserving the original semantics, and applies an adaptive counterfactual filling strategy to replace the masked regions without introducing external semantics. Feeding these counterfactual samples back into training encourages the model to exploit the remaining informative evidence and shifts the decision boundary toward a more robust one. Extensive experiments across five ImageNet variants show that SS-CA effectively improves ID accuracy, OOD generalization, and perturbation robustness, achieving gains of 5.70%/18.04% on ImageNet-1k/ImageNet-R with CLIP ViT/32b, 9.52%/11.33% on ImageNet-R/ImageNet-S on TinyImageNet-200 with ResNet-101, and about 4% under Gaussian Noise corruption. The code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。