通过注意力重分配提升视觉语言模型在开放词汇分割中的精准度
Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective
- 发现CLIP在分割时会分心到无关区域，导致注意力分散
- 提出RF-CLIP方法，训练零成本重定向注意力，提升对目标区域的聚焦
- 在8个基准上达到顶尖性能，适合追求高精度与效率的视觉分割研究者
开放词汇语义分割（OVSS）依赖像素级视觉-语言对齐，将类别提示与对应像素关联。关键挑战在于增强多模态密集预测能力，特别是像素级多模态对齐。尽管现有方法利用CLIP的视觉-语言对齐取得良好效果，但很少从可解释性机制角度探究其在密集预测中的性能边界。本文系统分析了CLIP内部机制，发现一种关键现象：类似于人类分心，CLIP将大量注意力资源从目标区域转移到无关标记。分析表明，这些干扰项源于特定维度的过激活；过滤它们能显著提升CLIP的密集预测表现。为此，我们提出训练零成本的ReFocusing CLIP（RF-CLIP）方法，模拟人类分心-聚焦行为，将注意力从干扰标记重新引导至目标区域，从而细化多模态对齐粒度。该方法在八个基准上均实现最优性能，同时保持高效推理。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) employs pixel-level vision-language alignment to associate category-related prompts with corresponding pixels. A key challenge is enhancing the multimodal dense prediction capability, specifically this pixel-level multimodal alignment. Although existing methods achieve promising results by leveraging CLIP's vision-language alignment, they rarely investigate the performance boundaries of CLIP for dense prediction from an interpretability mechanisms perspective. In this work, we systematically investigate CLIP's internal mechanisms and identify a critical phenomenon: analogous to human distraction, CLIP diverts significant attention resources from target regions to irrelevant tokens. Our analysis reveals that these tokens arise from dimension-specific over-activation; filtering them enhances CLIP's dense prediction performance. Consequently, we propose ReFocusing CLIP (RF-CLIP), a training-free approach that emulates human distraction-refocusing behavior to redirect attention from distraction tokens back to target regions, thereby refining CLIP's multimodal alignment granularity. Our method achieves SOTA performance on eight benchmarks while maintaining high inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。