提出新任务REIR,实现跨图搜物体并准确定位。
Referring Expression Instance Retrieval and A Strong End-to-End Baseline
- 设计双流网络,文本与视觉分支协同定位目标
- 在MSCOCO和RefCOCO上构建高质量基准REIRCOCO
- 首次端到端支持大规模图像中细粒度实例检索
自然语言查询视觉信息是现实应用的基本需求。文本-图像检索(TIR)基于图像级描述从图库中检索目标图像,而指代表达理解(REC)则利用实例级描述在给定图像中定位目标对象。然而,实际场景常需在大型图库中通过细粒度描述查询特定实例,并同时获取相关图像及实例位置。此时,TIR难以处理细粒度描述与对象定位,而REC在大规模图库中搜索效率低且缺乏有效排序机制。本文提出新任务——指代表达实例检索(REIR),支持基于细粒度描述的实例级检索与定位。首先,我们构建了大规模基准REIRCOCO,通过先进视觉语言模型在MSCOCO和RefCOCO数据集上生成高质量指代表达。其次,提出端到端基线方法CLARE,采用双流架构:文本分支编码描述为查询嵌入,视觉分支检测候选对象并提取实例级特征;选择与查询最相似的候选进行边界框预测。CLARE先在物体检测与REC数据集上预训练建立语义对齐能力,再通过对比语言-实例对齐(CLIA)优化跨图像检索性能。代码与基准将公开。
原文摘要 · Abstract (English)
Using natural language to query visual information is a fundamental need in real-world applications. Text-Image Retrieval (TIR) retrieves a target image from a gallery based on an image-level description, while Referring Expression Comprehension (REC) localizes a target object within a given image using an instance-level description. However, real-world applications often present more complex demands. Users typically query an instance-level description across a large gallery and expect to receive both relevant image and the corresponding instance location. In such scenarios, TIR struggles with fine-grained descriptions and object-level localization, while REC is limited in its ability to efficiently search large galleries and lacks an effective ranking mechanism. In this paper, we introduce a new task called \textbf{Referring Expression Instance Retrieval (REIR)}, which supports both instance-level retrieval and localization based on fine-grained referring expressions. First, we propose a large-scale benchmark for REIR, named REIRCOCO, constructed by prompting advanced vision-language models to generate high-quality referring expressions for instances in the MSCOCO and RefCOCO datasets. Second, we present a baseline method, Contrastive Language-Instance Alignment with Relation Experts (CLARE), which employs a dual-stream architecture to address REIR in an end-to-end manner. Given a referring expression, the textual branch encodes it into a query embedding. The visual branch detects candidate objects and extracts their instance-level visual features. The most similar candidate to the query is selected for bounding box prediction. CLARE is first trained on object detection and REC datasets to establish initial grounding capabilities, then optimized via Contrastive Language-Instance Alignment (CLIA) for improved retrieval across images. We will release our code and benchmark publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。