用图像局部结构修复CLIP特征，提升无训练开放词汇分割精度
Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentation
- 构建区域邻接图捕捉局部结构关系，指导特征修正
- 在多个基准上显著降低分割噪声，提升区域一致性
- 适合追求高精度且无需微调的开放词汇分割场景
得益于大规模数据集学习到的归纳偏置，开放词汇语义分割（OVSS）利用视觉-语言模型（如CLIP）实现了显著进展，无需特定任务训练。然而，由于CLIP在图像-文本对上预训练，更关注全局语义对齐，导致其在细粒度视觉区域与文本关联时表现不佳，尤其在局部区域产生噪声和不一致预测。我们将其归因于对比学习范式带来的分散偏差，仅靠CLIP特征难以缓解。为此，我们提出一种结构感知特征修正方法，直接从图像中提取实例级先验。具体而言，基于低层特征（如颜色、纹理）构建区域邻接图（RAG），捕捉局部结构关系，并通过增强局部区分性来优化CLIP特征。大量实验表明，该方法有效抑制分割噪声，提升区域一致性，在多个开放词汇分割基准上表现优异。
原文摘要 · Abstract (English)
Benefiting from the inductive biases learned from large-scale datasets, open-vocabulary semantic segmentation (OVSS) leverages the power of vision-language models, such as CLIP, to achieve remarkable progress without requiring task-specific training. However, due to CLIP's pre-training nature on image-text pairs, it tends to focus on global semantic alignment, resulting in suboptimal performance when associating fine-grained visual regions with text. This leads to noisy and inconsistent predictions, particularly in local areas. We attribute this to a dispersed bias stemming from its contrastive training paradigm, which is difficult to alleviate using CLIP features alone. To address this, we propose a structure-aware feature rectification approach that incorporates instance-specific priors derived directly from the image. Specifically, we construct a region adjacency graph (RAG) based on low-level features (e.g., colour and texture) to capture local structural relationships and use it to refine CLIP features by enhancing local discrimination. Extensive experiments show that our method effectively suppresses segmentation noise, improves region-level consistency, and achieves strong performance on multiple open-vocabulary segmentation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。