让图像检索聚焦关键属性,提升精准度。
Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
- 用可提示的嵌入突出查询所需的视觉属性
- 在COVO-Facet上实现Recall@5提升15%和8%
- 适用于需要细粒度检索的应用场景
一张图胜过千言万语,但只有少数信息对特定任务至关重要。理想的文本到图像检索器应聚焦于与查询相关的特定视觉属性。为评估现有检索器在属性聚焦查询上的表现,我们构建了基于COCO的基准COVO-Facet,包含9,112个关于多样属性的查询。发现主流的CLIP类检索器因关注全局语义和主体而忽略细节,性能差且不平衡;甚至最新基于多模态大模型(MLLM)的更强检索器也存在此问题。因此我们提出使用由多模态检索器生成的可提示图像嵌入,通过强调所需属性提升性能。该方法可泛化至不同查询类型、图像池及基础架构。为增强实用性,我们提供两种加速策略:预处理可提示嵌入与线性近似。前者在预定义提示下使Recall@5提升15%,后者在推理时仅获提示条件下实现8%提升。
原文摘要 · Abstract (English)
While an image is worth more than a thousand words, only a few provide crucial information for a given task and thus should be focused on. In light of this, ideal text-to-image (T2I) retrievers should prioritize specific visual attributes relevant to queries. To evaluate current retrievers on handling attribute-focused queries, we build COCO-Facet, a COCO-based benchmark with 9,112 queries about diverse attributes of interest. We find that CLIP-like retrievers, which are widely adopted due to their efficiency and zero-shot ability, have poor and imbalanced performance, possibly because their image embeddings focus on global semantics and subjects while leaving out other details. Notably, we reveal that even recent Multimodal Large Language Model (MLLM)-based, stronger retrievers with a larger output dimension struggle with this limitation. Hence, we hypothesize that retrieving with general image embeddings is suboptimal for performing such queries. As a solution, we propose to use promptable image embeddings enabled by these multimodal retrievers, which boost performance by highlighting required attributes. Our pipeline for deriving such embeddings generalizes across query types, image pools, and base retriever architectures. To enhance real-world applicability, we offer two acceleration strategies: Pre-processing promptable embeddings and using linear approximations. We show that the former yields a 15% improvement in Recall@5 when prompts are predefined, while the latter achieves an 8% improvement when prompts are only available during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。