arXiv:2504.10995cs.CVcs.AI2025-04被引 5

TMCIR通过动态融合图文信息,更准确捕捉用户修改意图。

TMCIR: Token Merge Benefits Composed Image Retrieval

  • 用扩散模型生成伪目标图,增强文本对意图的感知能力
  • 自适应融合视觉与文本特征,在Fashion-IQ上达83.6%精度
  • 适合需要精准图文联合检索的场景,如时尚商品定制

组合图像检索(CIR)利用包含参考图像和文本描述修改需求的多模态查询来检索目标图像。当前跨模态特征融合方法在意图理解上存在固有偏差,过度侧重参考图像(视觉主导)或文本意图(文本主导)。为此,我们提出TMCIR框架,包含两项创新:1)意图感知的跨模态对齐——通过扩散模型合成由参考图像与文本描述生成的伪目标图像,对CLIP编码器进行对比微调,提升文本捕捉细微意图的能力;2)自适应标记融合——通过对比学习微调所有编码器,将自适应融合特征与真实目标图像对比,动态平衡视觉与文本表示。在Fashion-IQ和CIRR数据集上的实验表明,TMCIR显著优于现有方法,尤其在捕捉细微用户意图方面表现突出。

原文摘要 · Abstract (English)

Composed Image Retrieval (CIR) retrieves target images using a multi-modal query that combines a reference image with text describing desired modifications. The primary challenge is effectively fusing this visual and textual information. Current cross-modal feature fusion approaches for CIR exhibit an inherent bias in intention interpretation. These methods tend to disproportionately emphasize either the reference image features (visual-dominant fusion) or the textual modification intent (text-dominant fusion through image-to-text conversion). Such an imbalanced representation often fails to accurately capture and reflect the actual search intent of the user in the retrieval results. To address this challenge, we propose TMCIR, a novel framework that advances composed image retrieval through two key innovations: 1) Intent-Aware Cross-Modal Alignment. We first fine-tune CLIP encoders contrastively using intent-reflecting pseudo-target images, synthesized from reference images and textual descriptions via a diffusion model. This step enhances the encoder ability of text to capture nuanced intents in textual descriptions. 2) Adaptive Token Fusion. We further fine-tune all encoders contrastively by comparing adaptive token-fusion features with the target image. This mechanism dynamically balances visual and textual representations within the contrastive learning pipeline, optimizing the composed feature for retrieval. Extensive experiments on Fashion-IQ and CIRR datasets demonstrate that TMCIR significantly outperforms state-of-the-art methods, particularly in capturing nuanced user intent.

图像检索跨模态扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。