用结构对齐提升DINO的边界感知,实现更精准的开放词汇分割。
OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance
- 通过结构对齐激活DINO深层特征的边缘敏感性。
- 在多个弱监督基准上平均提升2.1%,复杂场景提升6.3%。
- 适合需要高精度分割的开放词汇视觉任务研究者。
开放词汇分割(OVS)旨在基于语义描述分割预定义类别之外的图像区域。尽管基于CLIP的方法在语义泛化上表现优异,但常缺乏细粒度空间感知能力。近期工作引入了如DINO等视觉基础模型以缓解该问题,但仍难以精确捕捉边界信息。本文分析DINO内部表征,发现其边界感知能力并非缺失,而是随深度变换块逐步衰减。为此,提出OVS-DINO框架,通过与段落任意模型(SAM)的结构对齐,重激活DINO的潜在边缘敏感性。具体设计结构感知编码器(SAE)和结构调制解码器(SMD),利用SAM的结构先验激活边界特征,并结合SAM生成伪掩码进行监督。大量实验表明,本方法在多个弱监督OVS基准上达到当前最优性能,平均分数提升2.1%(从44.8%到46.9%)。尤其在复杂杂乱场景中,于Cityscapes数据集上提升6.3%(从36.6%到42.9%)。
原文摘要 · Abstract (English)
Open-Vocabulary Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。