无需训练即可实现文本指定风格的图像内容检索
Composed Image Retrieval for Training-Free Domain Conversion
- 用词典最近邻搜索实现图像到文本的离散映射
- 在标准和新基准上显著超越已有方法
- 适合需要快速适配新领域的视觉生成任务
本文研究无需训练的域转换中的组合图像检索问题，即根据查询文本从目标域中检索与查询图像内容一致的图像。我们发现，强大的视觉语言模型本身具备充分描述能力，无需额外训练。通过文本反演技术将查询图像映射至文本输入空间，不同于常规在连续文本标记空间进行反演，我们采用离散词汇空间，并通过文本词典的最近邻搜索实现映射。该方法使图像在词汇空间中软性传递，并借助基于检索的增强提升鲁棒性。数据库图像通过加权集成多个文本查询（包括映射词与域描述词）进行检索。在标准及新提出的基准测试中，本方法大幅优于现有技术。代码已开源。
原文摘要 · Abstract (English)
This work addresses composed image retrieval in the context of domain conversion, where the content of a query image is retrieved in the domain specified by the query text. We show that a strong vision-language model provides sufficient descriptive power without additional training. The query image is mapped to the text input space using textual inversion. Unlike common practice that invert in the continuous space of text tokens, we use the discrete word space via a nearest-neighbor search in a text vocabulary. With this inversion, the image is softly mapped across the vocabulary and is made more robust using retrieval-based augmentation. Database images are retrieved by a weighted ensemble of text queries combining mapped words with the domain text. Our method outperforms prior art by a large margin on standard and newly introduced benchmarks. Code: https://github.com/NikosEfth/freedom
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。