arXiv:2608.07886cs.CVcs.AI2026-08

提出双向概念对应框架,让模型自动找图文匹配部分。

Vision-Language Grounding as Bidirectional Concept Correspondence

论文配图:Vision-Language Grounding as Bidirectional Concept Correspondence
图 1 · 摘自论文原文
  • 将图文定位转为双向对应预测,不预设文本位置
  • 在长描述数据集上对应率提升48%,零样本LVIS提升29%
  • 统一了短语定位、指代表达等任务,适合多模态理解场景

视觉-语言定位通常被简化为单向定位问题:给定一个已知的文本短语或类别名,找出其对应的图像区域。这种设定假设相关语言单元已知,忽略了基础的指代难题——即哪些文本部分具有视觉指代性,以及如何与图像中的实体对应。本文将定位建模为图像-文本对之间的双向概念对应问题:给定一对图像和文本,目标是恢复所有视觉可指代的文本片段与图像实例级区域之间的对应关系,无需预先提供相关文本片段。该框架统一了常见定位任务,包括短语定位、指代表达定位和开放词汇检测,将文本分割、图像分割与跨模态对齐视为单一对应预测问题。为此,我们提出基于预训练视觉-语言模型的ConCor-1模型,利用可学习的桥接标记表示候选对应关系,并预测每个标记的文本掩码、图像掩码和对应存在分数。为训练与评估该任务,我们将多种定位与分割数据集转换为统一的对应格式。实验表明,ConCor-1持续优于基线,在长描述数据集上对应率提升48%,在零样本LVIS上提升29%(大类别列表作为文本输入)。

原文摘要 · Abstract (English)

Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as $\textit{bidirectional concept correspondence}$ over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce $\textbf{ConCor-1}$, a grounding model built on top of a pretrained vision-language model. It uses learnable $\textit{bridge tokens}$ to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that $\textbf{ConCor-1}$ consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.

视觉语言图文对应零样本检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。