构建首个统一图像检索与精确定位的中文论文数据集
ReSeDis: A Dataset for Referring-based Object Search across Large-Scale Image Collections
- 提出新任务ReSeDis,同时实现跨大规模图像集的检索与像素级定位
- 在包含数万张图像的基准数据集上,首次实现描述与目标实例一一对应
- 提供零样本基线模型,为多模态搜索系统研究留出巨大提升空间
大型视觉搜索引擎需同时解决两个问题:(i)找出所有包含某句描述对象的图像;(ii)在每张命中图像中准确定位该对象的边界框或像素位置。现有方法仅解决其中一端:视觉定位虽可生成精确框和掩码,但假设对象必存在于每张测试图像,导致在网页规模数据集中产生大量误报;文本到图像检索擅长从海量数据库中排序相关图像,但仅返回整图匹配,无法进行细粒度定位。为此,我们提出引用式对象搜索与发现(ReSeDis),首个将语料库级检索与像素级定位统一的任务。给定自由描述,模型需判断查询对象是否出现在每张图像中,并在存在时返回其边界框或分割掩码。为支持严谨研究,我们构建了一个基准数据集,其中每个描述唯一对应散布于大规模、多样化语料库中的对象实例,避免了无意匹配。我们还设计了任务专用指标,联合评估检索召回率与定位精度。最后,我们提供基于冻结视觉-语言模型的简单零样本基线,揭示未来研究的巨大潜力。ReSeDis为构建下一代鲁棒、可扩展的多模态搜索系统提供了真实、端到端的测试平台。
原文摘要 · Abstract (English)
Large-scale visual search engines are expected to solve a dual problem at once: (i) locate every image that truly contains the object described by a sentence and (ii) identify the object's bounding box or exact pixels within each hit. Existing techniques address only one side of this challenge. Visual grounding yields tight boxes and masks but rests on the unrealistic assumption that the object is present in every test image, producing a flood of false alarms when applied to web-scale collections. Text-to-image retrieval excels at sifting through massive databases to rank relevant images, yet it stops at whole-image matches and offers no fine-grained localization. We introduce Referring Search and Discovery (ReSeDis), the first task that unifies corpus-level retrieval with pixel-level grounding. Given a free-form description, a ReSeDis model must decide whether the queried object appears in each image and, if so, where it is, returning bounding boxes or segmentation masks. To enable rigorous study, we curate a benchmark in which every description maps uniquely to object instances scattered across a large, diverse corpus, eliminating unintended matches. We further design a task-specific metric that jointly scores retrieval recall and localization precision. Finally, we provide a straightforward zero-shot baseline using a frozen vision-language model, revealing significant headroom for future study. ReSeDis offers a realistic, end-to-end testbed for building the next generation of robust and scalable multimodal search systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。