用两阶段注意力机制过滤干扰背景，提升模型鲁棒性
Two-stage Vision Transformers and Hard Masking offer Robust Object Representations
- 分两阶段：先用全图找关键区域，再聚焦这些区域进行分析
- 在多个基准上显著降低对异常背景的错误依赖
- 可解释的二值掩码支持测试时干预，适合需要可靠推理的场景
上下文会强烈影响物体表征，可能导致不良偏差，尤其在推理时物体出现在分布外背景中。许多以物体为中心的任务需要利用上下文来识别相关图像区域，但上下文也可能带来干扰。我们提出一种基于注意力的方法，通过学习得到的二值注意力掩码，确保只有被关注的图像区域影响预测结果。为此，我们评估了一种两阶段框架：第一阶段处理整幅图像，发现物体部件并识别任务相关区域（此时需依赖上下文）；第二阶段使用输入注意力掩码，将感受野限制在这些区域，实现专注分析并过滤潜在无关信息。两个阶段联合训练，使第二阶段能优化第一阶段。显式的语义掩码也使模型推理过程可审计，支持强大的测试时干预以进一步提升鲁棒性。在多个多样化基准上的大量实验表明，该方法显著增强了对虚假关联和分布外背景的鲁棒性。
原文摘要 · Abstract (English)
Context can strongly affect object representations, sometimes leading to undesired biases, particularly when objects appear in out-of-distribution backgrounds at inference. At the same time, many object-centric tasks require to leverage the context for identifying the relevant image regions. We posit that this conundrum, in which context is simultaneously needed and a potential nuisance, can be addressed by an attention-based approach that uses learned binary attention masks to ensure that only attended image regions influence the prediction. To test this hypothesis, we evaluate a two-stage framework: stage 1 processes the full image to discover object parts and identify task-relevant regions, for which context cues are likely to be needed, while stage 2 leverages input attention masking to restrict its receptive field to these regions, enabling a focused analysis while filtering out potentially spurious information. Both stages are trained jointly, allowing stage 2 to refine stage 1. The explicit nature of the semantic masks also makes the model's reasoning auditable, enabling powerful test-time interventions to further enhance robustness. Extensive experiments across diverse benchmarks demonstrate that this approach significantly improves robustness against spurious correlations and out-of-distribution backgrounds. Code: https://github.com/ananthu-aniraj/ifam
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。