让视觉Transformer更聚焦物体，减少背景干扰
Inhibited Self-Attention: Sharpening Focus in Vision Transformers
- 引入抑制信号，用负注意力值压制无关特征
- 在ImageNet-1k和COCO上提升对象中心选择性
- 适合关注模型可解释性和鲁棒性的研究者
视觉变压器（ViTs）在计算机视觉任务中表现卓越，但其自注意力机制常将关注力分散到背景区域，依赖表面相关性而非对象相关线索。受生物视觉系统中抑制机制启发，我们提出抑制性自注意力（ISA），一种新自注意力机制，通过整合抑制信号增强特征选择性并抑制虚假响应。与传统自注意力仅依赖softmax归一化的正注意力值不同，我们的方法保留并利用负注意力分数，以抑制无关特征并强化对感兴趣对象的关注。在ImageNet-1k、COCO等多个数据集及多个鲁棒性基准上的实验表明，ISA提升了对象中心选择性，减少了对捷径的依赖，并改善了分布外泛化能力。对相关性图的分析确认，使用ISA的ViTs在对象相关区域展现出更锐利、更局部的聚焦，同时减少来自非相关（背景）特征的干扰，从而实现更可靠的模型。代码已公开于https://github.com/prdvanderwal/inhibited-self-attention。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。