让文字找图更准更快，还能定位具体物体。
Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval
- 分两阶段：先用掩码和嵌入快速筛选图像，再用大模型精修定位。
- 在COCO和D³数据集上，检索准确率与分割质量均显著提升。
- 适合需要精准图文匹配与对象定位的场景，如智能搜索、视觉问答。
文本到图像检索（TIR）旨在根据文本查询找到相关图像，但现有方法主要依赖整图描述，缺乏可解释性。与此同时，指代表达分割（RES）能基于自然语言精确定位物体，但在大规模图像集合中计算成本高。为弥合这一差距，我们提出掩码感知的文本到图像检索（MaTIR），将TIR与RES统一，要求兼具高效的图像搜索和精准的物体分割。为此，我们设计一个两阶段框架：第一阶段进行掩码感知的图像检索，利用SAM 2生成物体掩码，通过Alpha-CLIP离线提取区域级嵌入，实现高效可扩展的在线检索；第二阶段使用多模态大语言模型（MLLM）重排结果并生成边界框，与掩码匹配以完成对象定位。我们在COCO和D³数据集上评估该方法，结果表明在检索准确率与分割质量上均显著优于此前方法。
原文摘要 · Abstract (English)
Text-to-image retrieval (TIR) aims to find relevant images based on a textual query, but existing approaches are primarily based on whole-image captions and lack interpretability. Meanwhile, referring expression segmentation (RES) enables precise object localization based on natural language descriptions but is computationally expensive when applied across large image collections. To bridge this gap, we introduce Mask-aware TIR (MaTIR), a new task that unifies TIR and RES, requiring both efficient image search and accurate object segmentation. To address this task, we propose a two-stage framework, comprising a first stage for segmentation-aware image retrieval and a second stage for reranking and object grounding with a multimodal large language model (MLLM). We leverage SAM 2 to generate object masks and Alpha-CLIP to extract region-level embeddings offline at first, enabling effective and scalable online retrieval. Secondly, MLLM is used to refine retrieval rankings and generate bounding boxes, which are matched to segmentation masks. We evaluate our approach on COCO and D$^3$ datasets, demonstrating significant improvements in both retrieval accuracy and segmentation quality over previous methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。