先生成图像再检索,提升图文搜索准确率
VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval
- 用文本生成图像,再在图像域内检索,避开跨模态匹配缺陷
- 在四个基准上平均提升nDCG@30达0.125,多实体任务增益高达15.7%
- 适配多种生成模型与下游问答系统,易复现且效果显著
文本到图像检索(T2I retrieval)仍具挑战性,因跨模态嵌入常表现为概念集合,难以表征姿态、视角等结构化视觉关系。我们提出「可视化后检索」(VisRet)范式,通过文本生成图像,再在图像域内进行检索,规避跨模态检索器对细微视觉空间特征识别的不足。在四个基准(Visual-RAG、INQUIRE-Rerank、Microsoft COCO及新提出的Visual-RAG-ME多实体对比数据集)上,VisRet显著优于跨模态相似度匹配方法及将T2I检索重构成文本到文本相似度匹配的基线,使用CLIP作为检索器时,nDCG@30平均提升0.125;使用E5-V时提升0.121。在下游问答任务中,VisRet使Visual-RAG和Visual-RAG-ME的top-1检索准确率分别提升3.8%和15.7%,top-10分别提升3.9%和11.1%。消融实验表明其兼容不同T2I指令大模型、生成模型及下游大模型。代码与新基准已公开于https://github.com/xiaowu0162/Visualize-then-Retrieve。
原文摘要 · Abstract (English)
Text-to-image retrieval (T2I retrieval) remains challenging because cross-modal embeddings often behave as bags of concepts, underrepresenting structured visual relationships such as pose and viewpoint. We proposeVisualize-then-Retrieve (VisRet), a retrieval paradigm that mitigates this limitation of cross-modal similarity alignment. VisRet first projects textual queries into the image modality via T2I generation, then performs retrieval within the image modality to bypass the weaknesses of cross-modal retrievers in recognizing subtle visual-spatial features. Across four benchmarks (Visual-RAG, INQUIRE-Rerank, Microsoft COCO, and our new Visual-RAG-ME featuring multi-entity comparisons), VisRet substantially outperforms cross-modal similarity matching and baselines that recast T2I retrieval as text-to-text similarity matching, improving nDCG@30 by 0.125 on average with CLIP as the retriever and by 0.121 with E5-V. For downstream question answering, VisRet increases accuracy on Visual-RAG and Visual-RAG-ME by 3.8% and 15.7% in top-1 retrieval, and by 3.9% and 11.1% in top-10 retrieval. Ablation studies show compatibility with different T2I instruction LLMs, T2I generation models, and downstream LLMs. VisRet provides a simple yet effective perspective for advancing in text-image retrieval. Our code and the new benchmark are publicly available at https://github.com/xiaowu0162/Visualize-then-Retrieve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。