用大模型生成实体视觉描述,提升图文检索准确率
EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models
- 用大模型生成实体视觉描述,补充文本查询的视觉信息
- 在多个基准上显著提升图文检索效果,最高提升6.2个点
- 适合需要精准图文匹配的应用场景,如智能搜索
视觉-语言检索(VLR)在学术界和工业界均受到广泛关注,涉及以文本(或图像)为查询检索对应图像(或文本)。然而,现有方法常忽略实体的丰富视觉语义知识,导致检索结果错误。为此,我们提出实体视觉描述增强的CLIP(EvdCLIP),旨在利用实体的视觉知识来丰富查询。具体而言,由于人类通过视觉线索识别实体,我们采用大语言模型(LLM)生成实体视觉描述(EVDs)作为对齐线索,以补充文本数据。这些EVDs被整合到原始查询中,形成富含视觉信息的增强查询。此外,考虑到增强查询可能引入噪声或低质量扩展,我们设计了一种可训练的EVD感知重写器(EaRW),利用EVD知识和语言模型的生成能力有效重写查询。通过专门的训练策略,EaRW能够生成高质量、低噪声的增强查询。在多个图像-文本检索基准上的大量定量与定性实验验证了EvdCLIP在视觉-语言检索任务中的优越性。
原文摘要 · Abstract (English)
Vision-language retrieval (VLR) has attracted significant attention in both academia and industry, which involves using text (or images) as queries to retrieve corresponding images (or text). However, existing methods often neglect the rich visual semantics knowledge of entities, thus leading to incorrect retrieval results. To address this problem, we propose the Entity Visual Description enhanced CLIP (EvdCLIP), designed to leverage the visual knowledge of entities to enrich queries. Specifically, since humans recognize entities through visual cues, we employ a large language model (LLM) to generate Entity Visual Descriptions (EVDs) as alignment cues to complement textual data. These EVDs are then integrated into raw queries to create visually-rich, EVD-enhanced queries. Furthermore, recognizing that EVD-enhanced queries may introduce noise or low-quality expansions, we develop a novel, trainable EVD-aware Rewriter (EaRW) for vision-language retrieval tasks. EaRW utilizes EVD knowledge and the generative capabilities of the language model to effectively rewrite queries. With our specialized training strategy, EaRW can generate high-quality and low-noise EVD-enhanced queries. Extensive quantitative and qualitative experiments on image-text retrieval benchmarks validate the superiority of EvdCLIP on vision-language retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。