解决图像分割中窗口独立处理导致的语义不一致问题
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
- 跨窗口扩展注意力机制,融合全局上下文信息
- 引入代理锚点统一衡量内外窗口的相似性,缓解窗口偏差
- 动态归一化提升小目标分割性能,兼容现有方法
近期无需训练的开放词汇语义分割方法普遍采用滑动窗口推理策略以克服CLIP处理高分辨率图像的局限。然而,该策略导致各窗口独立处理,引发窗口间语义不一致问题。为此,本文提出全局-局部对齐的CLIP(GLA-CLIP)框架,促进窗口间全面信息交换。不同于仅关注单个窗口内的注意力,GLA-CLIP将键值令牌扩展至包含所有窗口的上下文线索。同时观察到窗口偏差:外窗令牌较难被关注,因查询特征仅由内窗块间交互生成,缺乏超出局部上下文的语义支撑。为此,我们引入一个代理锚点,通过聚合所有窗口中与给定查询高度相似的令牌,为内、外窗块提供统一的语义参考。此外,提出动态归一化方案,根据物体尺度动态缩放并阈值化注意力图,以应对小目标场景。实验表明,GLA-CLIP可显著提升无需训练的开放词汇语义分割性能,且能部署于现有方法以扩大感受野。代码已开源。
原文摘要 · Abstract (English)
A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at https://github.com/2btlFe/GLA-CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。