用大模型集体推理统一多种图像检索任务,提升准确率。
ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning
- 通过思维链生成多粒度搜索意图,增强鲁棒性
- 分阶段验证谓词与成对评估,提升结果可靠性
- 无需训练,适配多种语言引导图像检索场景
随着在线内容中图像的激增,语言引导图像检索(LGIR)在过去十年成为研究热点,涵盖多种输入形式的子任务。尽管大模态模型(LMMs)显著推动了这些任务的发展,但现有方法常孤立处理各任务,需为每项任务构建独立系统,不仅增加复杂性和维护成本,还加剧了语言歧义和图像内容复杂性带来的挑战,导致检索结果不准确。为此,我们提出 ImageScope,一个无需训练的三阶段框架,利用集体推理统一 LGIR 任务。其核心思想在于语言的组合性,将多样化的 LGIR 任务转化为通用的文本到图像检索过程,并由 LMM 推理作为通用验证机制优化结果。具体而言,第一阶段通过思维链(CoT)推理在不同语义粒度上合成搜索意图,提升鲁棒性;第二、三阶段分别进行局部谓词验证和全局成对评估。在六个 LGIR 数据集上的实验表明,ImageScope 显著优于基线方法。全面评估与消融研究进一步验证了设计的有效性。
原文摘要 · Abstract (English)
With the proliferation of images in online content, language-guided image retrieval (LGIR) has emerged as a research hotspot over the past decade, encompassing a variety of subtasks with diverse input forms. While the development of large multimodal models (LMMs) has significantly facilitated these tasks, existing approaches often address them in isolation, requiring the construction of separate systems for each task. This not only increases system complexity and maintenance costs, but also exacerbates challenges stemming from language ambiguity and complex image content, making it difficult for retrieval systems to provide accurate and reliable results. To this end, we propose ImageScope, a training-free, three-stage framework that leverages collective reasoning to unify LGIR tasks. The key insight behind the unification lies in the compositional nature of language, which transforms diverse LGIR tasks into a generalized text-to-image retrieval process, along with the reasoning of LMMs serving as a universal verification to refine the results. To be specific, in the first stage, we improve the robustness of the framework by synthesizing search intents across varying levels of semantic granularity using chain-of-thought (CoT) reasoning. In the second and third stages, we then reflect on retrieval results by verifying predicate propositions locally, and performing pairwise evaluations globally. Experiments conducted on six LGIR datasets demonstrate that ImageScope outperforms competitive baselines. Comprehensive evaluations and ablation studies further confirm the effectiveness of our design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。