修复CLIP图像块相关性，提升开放词汇语义分割精度
CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation
- 用SAM界定块间交互范围，降低跨类相关性
- 引入自监督模型计算相干相似度，抑制错误关联
- 增强空间细节与语义表征，适合细粒度分割任务
开放词汇语义分割旨在无需预定义类别即可为每个像素分配语义标签。尽管对比语言-图像预训练（CLIP）在零样本分类上表现优异，但其图像块与类别嵌入对齐能力受限于块间相关性的不连贯性。本研究揭示，跨类别相关性是导致CLIP分割性能下降的主要原因。为此提出CorrCLIP，通过重构块相关性的范围与数值来改善。具体而言，CorrCLIP利用段落一切模型（SAM）定义块间交互范围，减少跨类相关性；针对SAM生成掩码可能包含多类块的问题，引入自监督模型计算一致的相似度值，压制跨类相关性的权重。此外，增加两个分支以强化块特征的空间细节和语义表达，并使用SAM生成的掩码更新分割图，提升空间一致性。基于对块相关性、特征表示和分割图的综合优化，CorrCLIP在八个基准测试中均取得更优性能。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation aims to assign semantic labels to each pixel without being constrained by a predefined set of categories. While Contrastive Language-Image Pre-training (CLIP) excels in zero-shot classification, it struggles to align image patches with category embeddings because of its incoherent patch correlations. This study reveals that inter-class correlations are the main reason for impairing CLIP's segmentation performance. Accordingly, we propose CorrCLIP, which reconstructs the scope and value of patch correlations. Specifically, CorrCLIP leverages the Segment Anything Model (SAM) to define the scope of patch interactions, reducing inter-class correlations. To mitigate the problem that SAM-generated masks may contain patches belonging to different classes, CorrCLIP incorporates self-supervised models to compute coherent similarity values, suppressing the weight of inter-class correlations. Additionally, we introduce two additional branches to strengthen patch features' spatial details and semantic representation. Finally, we update segmentation maps with SAM-generated masks to improve spatial consistency. Based on the improvement across patch correlations, feature representations, and segmentation maps, CorrCLIP achieves superior performance across eight benchmarks. Codes are available at: https://github.com/zdk258/CorrCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。