arXiv:2502.06818cs.LG2025-02被引 4

让CLIP在无需训练的情况下更好理解图像全局语义,提升分割精度。

Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

  • 重塑最后层注意力与值嵌入,融合全局语义信息增强特征表达
  • 在五个基准上超越现有最佳方法,平均提升2.1% mIoU
  • 适合研究开放词汇语义分割、想提升CLIP视觉理解能力的开发者

近期工作通过无训练方式改造CLIP实现开放词汇语义分割(TF-OVSS)。原始CLIP中,局部图像特征主要编码整体图像属性,不利于密集预测任务。以往方法为增强局部性,限制每个像素仅关注自身或邻近区域,导致全局上下文信息严重丢失。本文重新思考CLIP所蕴含的全局知识,提出GCLIP模型,旨在提取并利用有益的全局信息。由于每块特征最终由注意力权重与值嵌入决定,我们重构最后一层注意力与值嵌入,将有用全局上下文融入最终特征。首先,通过融合全局令牌块的注意力与查询-查询注意力,赋予注意力图像级属性,同时避免跨块同质化模式;其次,设计新颖的通道抑制策略,使最后一层注意力模块的值嵌入更具语义相关性。在五个标准数据集上的大量实验表明,该方法持续优于现有最先进方法。

原文摘要 · Abstract (English)

Recent works modify CLIP to perform open-vocabulary semantic segmentation in a training-free manner (TF-OVSS). In vanilla CLIP, patch-wise image representations mainly encode homogeneous image-level properties, which hinders the application of CLIP to the dense prediction task. Previous TF-OVSS works sacrifice globality to enhance the locality of CLIP features, by making each patch mainly attend to itself or its neighboring patches within a narrow local window. With their modifications,the ability of CLIP to aggregate global context information is largely weakened. Differently, in this paper, we rethink the global knowledge encoded by CLIP and propose GCLIP to answer how to extract and utilize beneficial global knowledge of CLIP for TF-OVSS. As the representation of each patch is finally determined by the attention weights and the Value embeddings, we propose to reshape the last-block attention and Value embeddings to aggregate useful global context into final features. Firstly, we aim to equip the last-block attention with image-level properties while not introducing homogeneous attention patterns across patches. To realize the goal, we fuse the attention from the global-token emerging blocks with the Query-Query attention. Secondly, we aim to make Value embeddings of the last-block attention module more semantically correlated. To realize this, we design a novel channel suppression strategy.Extensive experiments on five standard benchmarks demonstrate that our method consistently outperforms previous state-of-the-arts.

语义分割CLIP开放词汇全局信息

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。