用图像实例生成文本标记，实现个性化图文检索
Personalizing Retrieval using Joint Embeddings or "the Return of Fluffy"
- 设计可训练映射网络将物体图像嵌入转为文本标记
- 在两个基准上超越现有最佳性能，提升显著
- 适合需要精准人物/物体检索的应用场景
本文旨在通过结合图像中具体物体实例信息与自然语言描述，实现复合查询的图像检索。例如，检索一张「指定图像中的独角兽Fluffy在某人头上」的图片。为此，我们设计了一个映射网络（pi-map），能将局部图像嵌入（物体实例）转换为文本标记，使该标记与自然语言查询组合后，适用于CLIP式文本编码和图像检索。该文本标记的生成仅需一次简单训练，针对每个物体实例独立完成。实验表明，使用可训练映射网络配合冻结的CLIP图文编码器，在两个评估个性化检索的基准上均达到当前最优性能。
原文摘要 · Abstract (English)
The goal of this paper is to be able to retrieve images using a compound query that combines object instance information from an image, with a natural text description of what that object is doing or where it is. For example, to retrieve an image of "Fluffy the unicorn (specified by an image) on someone's head". To achieve this we design a mapping network that can "translate" from a local image embedding (of the object instance) to a text token, such that the combination of the token and a natural language query is suitable for CLIP style text encoding, and image retrieval. Generating a text token in this manner involves a simple training procedure, that only needs to be performed once for each object instance. We show that our approach of using a trainable mapping network, termed pi-map, together with frozen CLIP text and image encoders, improves the state of the art on two benchmarks designed to assess personalized retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。