用扩散模型生成视觉查询，提升类别级图文检索效果
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
- 用扩散模型将文本查询转为视觉查询，缩小模态差距
- 融合多张生成图特征，相似度匹配性能显著提升
- 适合需要精准类别检索的视觉语言应用
本文研究针对语义类别描述的文本到图像检索。尽管视觉-语言模型（如CLIP）提供开集词汇解决方案，但其将文本与图像映射至表征空间中相距较远的位置，限制了检索性能。为此，我们提出两步法：首先利用生成式扩散模型将文本查询转换为视觉查询；随后通过视觉模型估计图像间相似性。此外，引入聚合网络将多张生成图像融合为单一向量表示，并融合跨模态相似度分数。本方法充分利用视觉编码器、视觉-语言模型及文生图模型的进展。大量实验表明，其在各类别检索任务中均优于仅依赖文本查询的方法。代码已开源：https://github.com/faixan-khan/cletir
原文摘要 · Abstract (English)
This work explores text-to-image retrieval for queries that specify or describe a semantic category. While vision-and-language models (VLMs) like CLIP offer a straightforward open-vocabulary solution, they map text and images to distant regions in the representation space, limiting retrieval performance. To bridge this modality gap, we propose a two-step approach. First, we transform the text query into a visual query using a generative diffusion model. Then, we estimate image-to-image similarity with a vision model. Additionally, we introduce an aggregation network that combines multiple generated images into a single vector representation and fuses similarity scores across both query modalities. Our approach leverages advancements in vision encoders, VLMs, and text-to-image generation models. Extensive evaluations show that it consistently outperforms retrieval methods relying solely on text queries. Source code is available at: https://github.com/faixan-khan/cletir
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。