不训练也不依赖文本描述,直接生成心理图像实现精准检索。
Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrieval
- 用大模型直接生成查询对应的心理图像,替代传统文本描述
- 在真实图像库中构建对应合成图像,缩小与心理图像的域差异
- 完全零样本无训练,性能超越现有方法,适合无标注数据场景
组成图像检索(CIR)任务是通过包含参考图像和修改文本的多模态查询,在数据库中检索目标图像。文本描述如何修改参考图像形成‘心理图像’,系统需据此找到目标图像。核心挑战在于该心理图像并不存在,仅由查询隐式定义。现有方法采用大语言多模态模型(LMM)生成文本描述,再用视觉语言模型(VLM)进行图文匹配。本文从第一性原理出发,直接生成心理图像以实现更准确匹配:使用LMM为给定多模态查询生成心理图像,并以此搜索目标图像。由于心理图像与真实图像存在合成到真实的域差距,我们还为数据库中每张真实图像生成对应的合成图像以促进匹配。由此构建一个‘平行宇宙’(paracosm),实现查询与数据库图像的直接匹配。该方法无需训练、纯零样本,显著优于现有零样本方法,在多个挑战性基准上达到当前最优性能。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) is the task of retrieving a target image from a database using a multimodal query, which consists of a reference image and a modification text. The text specifies how to alter the reference image to form a ''mental image'', based on which CIR should find the target image in the database. The fundamental challenge of CIR is that this ''mental image'' is not physically available and is only implicitly defined by the query. The contemporary literature pursues zero-shot methods and uses a Large Multimodal Model (LMM) to generate a textual description for a given multimodal query, and then employs a Vision-Language Model (VLM) for textual-visual matching to search for the target image. In contrast, we address CIR from first principles by directly generating the ''mental image'' for more accurate matching. Particularly, we prompt an LMM to generate a ''mental image'' for a given multimodal query and propose to use this ''mental image'' to search for the target image. As the ''mental image'' has a synthetic-to-real domain gap with real images, we also generate a synthetic counterpart for each real image in the database to facilitate matching. In this sense, our method uses LMM to construct a ``paracosm'', where it matches the multimodal query and database images. Hence, we call this method Paracosm. Notably, Paracosm is a training-free zero-shot CIR method. It significantly outperforms existing zero-shot methods on challenging benchmarks, achieving state-of-the-art performance for zero-shot CIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。