arXiv:2506.13925cs.CVcs.AI2025-06被引 2

用语言锚点实现精准图像分割,小样本下表现远超现有方法

Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation

  • 将文本嵌入转为多尺度语义查询,增强上下文理解
  • 在少于1%标注数据下,多个数据集上超越当前最优结果
  • 适合做小样本、跨域图像分割任务的研究者参考

视觉语言模型(VLMs)虽具备丰富语义先验,但在半监督语义分割中仍被低估。现有方法忽视视觉与文本表征间的语义错位问题——因使用与领域无关的文本嵌入,未适配数据集和图像特定上下文。缺乏领域感知力与标注有限性,削弱了模型对语义的理解能力,导致上下文推理弱、类内区分差、易混淆相似类别。为此,我们提出分层视觉语言变换器(HVLFormer),在掩码变换器架构中实现视觉与文本表征的领域感知且鲁棒的对齐。首先,将预训练VLM的文本嵌入转化为文本对象查询,生成从粗到细粒度的多尺度、数据集感知查询,提升上下文推理能力。其次,通过注入图像特定视觉上下文,精炼查询以对齐文本语义与局部场景结构,增强类别区分能力。最后,引入跨视图与模态一致性正则化,确保在数据增强视图间预测一致,并在解码阶段保持稳定的视觉-语言对齐。仅需不足1%的训练数据,HVLFormer在Pascal VOC、COCO、ADE20K和Cityscapes上均优于现有最佳方法。代码与结果将在GitHub公开。

原文摘要 · Abstract (English)

Vision Language Models (VLMs) provide rich semantic priors but are underexplored in Semi supervised Semantic Segmentation. Recent attempts to integrate VLMs to inject high level semantics overlook the semantic misalignment between visual and textual representations that arises from using domain invariant text embeddings without adapting them to dataset and image specific contexts. This lack of domain awareness, coupled with limited annotations, weakens the model semantic understanding by preventing effective vision language alignment. As a result, the model struggles with contextual reasoning, shows weak intra class discrimination, and confuses similar classes. To address these challenges, we propose Hierarchical Vision Language transFormer (HVLFormer), which achieves domain aware and domain robust alignment between visual and textual representations within a mask transformer architecture. Firstly, we transform text embeddings from pretrained VLMs into textual object queries, enabling the generation of multi scale, dataset aware queries that capture class semantics from coarse to fine granularity and enhance contextual reasoning. Next, we refine these queries by injecting image specific visual context to align textual semantics with local scene structures and enhance class discrimination. Finally, to achieve domain robustness, we introduce cross view and modal consistency regularization, which enforces prediction consistency within mask-transformer architecture across augmented views. Moreover, it ensures stable vision language alignment during decoding. With less than 1% training data, HVLFormer outperforms state of the art methods on Pascal VOC, COCO, ADE20K, and Cityscapes. Our code and results will be available on GitHub.

图像分割视觉语言模型小样本学习半监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。