通过去噪映射图像到精准伪词,提升零样本图像检索效果
Denoise-I2W: Mapping Images to Denoising Words for Accurate Zero-Shot Composed Image Retrieval
- 构建伪三元组自动训练去噪映射网络
- 去除无关视觉信息,使伪词更聚焦操作意图
- 无需标注且兼容主流模型,性能提升1.45%~4.17%
零样本组合图像检索(ZS-CIR)支持多样化的视觉内容操作任务,涵盖领域、场景、物体和属性等。其核心挑战在于将图像表征准确映射为捕捉操作意图的伪词令牌。现有方法在检索与预训练阶段间存在伪词令牌冗余问题。本文提出一种新型去噪图像到词映射方法(Denoise-I2W),将图像映射为不含无关视觉信息的去噪伪词令牌,提升ZS-CIR准确性。首先,伪三元组构建模块自动生成伪三元组(即伪参考图像、伪操作文本和目标图像),用于预训练去噪映射网络;其次,伪组合映射模块将伪参考图像映射为伪词令牌,并与伪操作文本结合,使其对齐目标图像,从而消除无关视觉信息。Denoise-I2W为模型无关且无须标注的方法,在四个基准数据集上对三种先进ZS-CIR模型均表现出强泛化能力。集成后性能提升1.45%至4.17%,推理成本不变,达到新SOTA。代码已开源。
原文摘要 · Abstract (English)
Zero-Shot Composed Image Retrieval (ZS-CIR) supports diverse tasks with a broad range of visual content manipulation intentions that can be related to domain, scene, object, and attribute. A key challenge for ZS-CIR is to accurately map image representation to a pseudo-word token that captures the manipulation intention relevant image information for generalized CIR. However, existing methods between the retrieval and pre-training stages lead to significant redundancy in the pseudo-word tokens. In this paper, we propose a novel denoising image-to-word mapping approach, named Denoise-I2W, for mapping images into denoising pseudo-word tokens that, without intention-irrelevant visual information, enhance accurate ZS-CIR. Specifically, a pseudo triplet construction module first automatically constructs pseudo triples (\textit{i.e.,} a pseudo-reference image, a pseudo-manipulation text, and a target image) for pre-training the denoising mapping network. Then, a pseudo-composed mapping module maps the pseudo-reference image to a pseudo-word token and combines it with the pseudo-manipulation text with manipulation intention. This combination aligns with the target image, facilitating denoising intention-irrelevant visual information for mapping. Our proposed Denoise-I2W is a model-agnostic and annotation-free approach. It demonstrates strong generalization capabilities across three state-of-the-art ZS-CIR models on four benchmark datasets. By integrating Denoise-I2W with existing best models, we obtain consistent and significant performance boosts ranging from 1.45\% to 4.17\% over the best methods without increasing inference costs. and achieve new state-of-the-art results on ZS-CIR. Our code is available at \url{https://github.com/Pter61/denoise-i2w-tmm}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。