arXiv:2607.26107cs.CVcs.AI2026-07

不训练也不依赖外部模型,从CLIP中挖掘局部语义信息

TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions

论文配图:TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions
图 1 · 摘自论文原文
  • 通过分析CLS注意力输出,提取每块图像的语义贡献度
  • 在8个零样本分割任务上比现有方法提升1.3~4.5点mIoU
  • 适合希望无监督增强视觉语言模型局部理解的研究者

密集视觉-语言理解任务(如目标定位、区域识别、开放词汇语义分割)需要将语言概念与空间化的视觉区域关联。CLIP通过大规模对比预训练建立了共享的图文嵌入空间,但其图像级目标仅将文本与全局的CLS表示对齐,导致局部视觉-语言对应关系受到间接约束。现有方法多依赖额外标注、外部模型或特定任务微调,而无需训练的方法大多仅从已有图像块特征恢复密集响应,未探究局部语义在CLIP内部何时最易获取。我们提出TraceCLIP,一种无需训练的框架,通过分离写入CLS注意力输出中的块级特异性项,恢复潜在的块级语义证据。TraceCLIP进一步将贡献度生成的语义响应转化为语义测地拓扑门,校准最终层的块间亲和度以实现密集特征重建。诊断实验表明,这些贡献特征具备强局部语义区分能力与文本条件下的空间对齐性。在八个零样本语义分割基准上,TraceCLIP在两种主干网络和背景设置下,平均mIoU相较最强先验训练自由方法提升1.3至4.5点,且无需额外训练、外部视觉基础模型或区域级监督。更广泛地,这些发现提示:空间局部语义可能仍保留在全局对齐表示的内部结构中。

原文摘要 · Abstract (English)

Dense vision-language understanding, including object localization, region recognition, and open-vocabulary semantic segmentation, requires associating language concepts with spatially grounded visual regions. CLIP provides a strong foundation for these tasks by learning a shared image-text embedding space from large-scale contrastive pre-training. However, its image-level objective aligns text with a CLS-derived global representation, leaving local vision-language correspondence only indirectly constrained. Existing methods either introduce additional supervision, external models, or task-specific adaptation, while training-free approaches mainly recover dense responses from existing patch features without examining where local semantics become most accessible within CLIP. We introduce TraceCLIP, a training-free framework that recovers latent patch-level semantic evidence by isolating the patch-specific terms written into the CLS attention output. TraceCLIP further converts contribution-derived semantic responses into a semantic-geodesic topology gate that calibrates final-layer patch affinity for dense feature reconstruction. Diagnostic experiments show that these contribution features exhibit strong local semantic discrimination and text-conditioned spatial alignment. On eight zero-shot semantic segmentation benchmarks, TraceCLIP achieves gains of 1.3 to 4.5 points in average mIoU over the strongest prior training-free methods across both backbones and background settings, without additional training, external vision foundation models, or region-level supervision. More broadly, these findings suggest that spatially localized semantics may remain accessible within the internal construction of globally aligned representations.

CLIP语义分割零样本无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。