通过视觉语义互补融合,精准实现零样本图像修改检索。
Fine-Grained Zero-Shot Composed Image Retrieval with Complementary Visual-Semantic Integration
- 将参考图转为伪标记,结合修改描述与潜在新增对象
- 用多标签生成与大模型联合建模,完整捕捉语义变化
- 适合需要精细图像编辑的电商、设计场景
零样本图像组合检索(ZS-CIR)在实际应用中具有重要意义,允许用户通过参考图像和描述目标修改的相对文本检索目标图像。现有方法难以捕捉细粒度变化,且在视觉与语义信息融合上表现不佳,通常依赖图像转文本模型或大语言模型生成目标描述,但常忽略互补视觉信息与完整语义上下文。为此,我们提出细粒度零样本组合图像检索方法(CVSI),包含三个关键组件:(1) 视觉信息提取,不仅获取全局特征,还利用预训练映射网络将图像转换为伪标记,与修改文本及最可能添加的对象结合;(2) 语义信息提取,使用预训练描述模型为参考图像生成多个描述,再通过大语言模型生成修改后描述及可能添加的对象;(3) 互补信息检索,整合查询与数据库图像的信息,高效应对各类检索需求。在CIRR、CIRCO和FashionIQ三个公开数据集上的大量实验表明,CVSI显著优于现有最优方法。代码已开源:https://github.com/yyc6631/CVSI。
原文摘要 · Abstract (English)
Zero-shot composed image retrieval (ZS-CIR) is a rapidly growing area with significant practical applications, allowing users to retrieve a target image by providing a reference image and a relative caption describing the desired modifications. Existing ZS-CIR methods often struggle to capture fine-grained changes and integrate visual and semantic information effectively. They primarily rely on either transforming the multimodal query into a single text using image-to-text models or employing large language models for target image description generation, approaches that often fail to capture complementary visual information and complete semantic context. To address these limitations, we propose a novel Fine-Grained Zero-Shot Composed Image Retrieval method with Complementary Visual-Semantic Integration (CVSI). Specifically, CVSI leverages three key components: (1) Visual Information Extraction, which not only extracts global image features but also uses a pre-trained mapping network to convert the image into a pseudo token, combining it with the modification text and the objects most likely to be added. (2) Semantic Information Extraction, which involves using a pre-trained captioning model to generate multiple captions for the reference image, followed by leveraging an LLM to generate the modified captions and the objects most likely to be added. (3) Complementary Information Retrieval, which integrates information extracted from both the query and database images to retrieve the target image, enabling the system to efficiently handle retrieval queries in a variety of situations. Extensive experiments on three public datasets (e.g., CIRR, CIRCO, and FashionIQ) demonstrate that CVSI significantly outperforms existing state-of-the-art methods. Our code is available at https://github.com/yyc6631/CVSI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。