通过预测缺失视觉信息,提升零样本图像检索精度。
Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval
- 在隐空间中预测参考图缺失的目标内容,指导图像映射。
- 在6个任务上提升1.73%~4.45%,达到新最优性能。
- 适合需要精准跨域图像检索的研究者与应用开发者。
零样本组合图像检索(ZS-CIR)涉及跨领域、场景、物体和属性的多样化视觉内容操作任务。其核心挑战在于根据操作文本修改参考图像以准确检索目标图像,尤其当参考图像缺少关键目标内容时。本文提出一种基于预测的映射网络PrediCIR,可在隐空间中自适应预测参考图像缺失的目标视觉内容,从而实现精确的ZS-CIR。具体而言,世界视图生成模块首先通过省略目标视图的部分视觉内容,并结合来自现有图文对的操作意图,构建源视图;目标内容预测模块训练一个世界模型作为预测器,在隐空间中依据用户操作文本中的意图自适应预测缺失的视觉信息。两个模块共同将带有预测相关信息的图像映射为伪词令牌,无需额外监督。模型在六个ZS-CIR任务上展现出强泛化能力,相比最优方法性能提升1.73%至4.45%,并取得新的最佳结果。代码已开源:https://github.com/Pter61/predicir。
原文摘要 · Abstract (English)
Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent across domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to modify a reference image according to manipulation text to accurately retrieve a target image, especially when the reference image is missing essential target content. In this paper, we propose a novel prediction-based mapping network, named PrediCIR, to adaptively predict the missing target visual content in reference images in the latent space before mapping for accurate ZS-CIR. Specifically, a world view generation module first constructs a source view by omitting certain visual content of a target view, coupled with an action that includes the manipulation intent derived from existing image-caption pairs. Then, a target content prediction module trains a world model as a predictor to adaptively predict the missing visual information guided by user intention in manipulating text at the latent space. The two modules map an image with the predicted relevant information to a pseudo-word token without extra supervision. Our model shows strong generalization ability on six ZS-CIR tasks. It obtains consistent and significant performance boosts ranging from 1.73% to 4.45% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://github.com/Pter61/predicir.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。