用大模型自动生成数据,让图像检索不依赖人工标注。
Scaling Prompt Instructed Zero Shot Composed Image Retrieval with Image-Only Data

- 用大模型自动构建图文检索训练数据,无需人工标注。
- 在CIRR和FashionIQ上超越现有零样本方法,性能接近有监督模型。
- 仅靠图像数据即可训练,适合大规模无标注场景应用。
组合图像检索(CIR)旨在根据参考图像与自然语言描述的组合,检索出经过语义修改后的目标图像。传统CIR模型依赖包含参考图像、重述文本和目标图像的三元组数据,但这类数据需人工标注,成本高昂。尽管存在大量未标注图像数据,却难以规模化训练。本文提出新范式:利用大语言模型(LLM)自动生成三元组数据,仅依赖无标注图像集合。我们设计了嵌入重表述架构,有效融合图像与文本模态。所提模型InstructCIR在CIRR和FashionIQ数据集上,于零样本条件下超越当前最优方法。更重要的是,随着生成数据量增加,其性能逐步逼近有监督基线。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) is the task of retrieving images matching a reference image augmented with a text, where the text describes changes to the reference image in natural language. Traditionally, models designed for CIR have relied on triplet data containing a reference image, reformulation text, and a target image. However, curating such triplet data often necessitates human intervention, leading to prohibitive costs. This challenge has hindered the scalability of CIR model training even with the availability of abundant unlabeled data. With the recent advances in foundational models, we advocate a shift in the CIR training paradigm where human annotations can be efficiently replaced by large language models (LLMs). Specifically, we demonstrate the capability of large captioning and language models in efficiently generating data for CIR only relying on unannotated image collections. Additionally, we introduce an embedding reformulation architecture that effectively combines image and text modalities. Our model, named InstructCIR, outperforms state-of-the-art methods in zero-shot composed image retrieval on CIRR and FashionIQ datasets. Furthermore, we demonstrate that by increasing the amount of generated data, our zero-shot model gets closer to the performance of supervised baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。