arXiv:2603.19531cs.CVcs.AI2026-03

DINOv3.seg提升开放词汇语义分割精度,尤其在复杂场景下表现更佳。

dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3

  • 融合全局与局部视觉特征,增强语义与空间定位能力。
  • 通过早期和晚期双重优化图像-文本关联,提升密集预测准确性。
  • 适合需要高精度分割且类别动态扩展的视觉任务开发者。

开放词汇语义分割(OVSS)旨在为开放类别的文本定义类别分配像素级标签,要求模型在推理时对未见类别具备可靠泛化能力。尽管现代视觉语言模型(VLMs)支持强大的开放词汇识别,但其通过全局对比学习获得的表征在稠密预测任务上仍不理想,导致众多OVSS方法依赖有限的图像-文本相似度图适应或精炼,限制了复杂、杂乱场景下的空间精度与鲁棒性。本文提出DINOv3.seg,将DINOv3.txt扩展为专用于OVSS的框架。首先,设计面向该任务的专用架构,系统性地整合先前开放词汇分割工作的设计原则;其次,联合利用基于ViT编码器的全局[CLS] token与局部块级视觉特征所对应的文本嵌入,有效结合语义区分力与细粒度空间局部性;第三,不同于以往依赖后期相似度精炼的方法,我们提前对视觉表示进行精炼,再对生成的图像-文本相关特征进行后期精炼,从而在复杂场景中实现更准确、鲁棒的稠密预测;最后,提出基于滑动窗口聚合的高分辨率局部-全局推理策略,兼顾空间细节与全局上下文。我们在五个广泛采用的OVSS基准上进行了大量实验,结果表明该方法具有显著有效性与鲁棒性,持续优于当前最先进方法。

原文摘要 · Abstract (English)

Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference. Although modern vision-language models (VLMs) support strong open-vocabulary recognition, their representations learned through global contrastive objectives remain suboptimal for dense prediction, prompting many OVSS methods to depend on limited adaptation or refinement of image-text similarity maps. This, in turn, restricts spatial precision and robustness in complex, cluttered scenes. We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS. Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work. Second, we jointly leverage text embeddings aligned with both the global [CLS] token and local patch-level visual features from ViT-based encoder, effectively combining semantic discrimination with fine-grained spatial locality. Third, unlike prior approaches that rely primarily on post hoc similarity refinement, we perform early refinement of visual representations prior to image-text interaction, followed by late refinement of the resulting image-text correlation features, enabling more accurate and robust dense predictions in cluttered scenes. Finally, we propose a high-resolution local-global inference strategy based on sliding-window aggregation, which preserves spatial detail while maintaining global context. We conduct extensive experiments on five widely adopted OVSS benchmarks to evaluate our approach. The results demonstrate its effectiveness and robustness, consistently outperforming current state-of-the-art methods.

语义分割开放词汇视觉语言模型ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。