通过任务与模态适配,显著提升零样本图像检索效果
Modality and Task Adaptation for Enhanced Zero-shot Composed Image Retrieval
- 用大模型构建三元组数据,解决训练与推理目标不一致问题
- 引入轻量级适配器,使生成伪词更贴合下游文本编码器
- 适合想改进现有零样本检索方法的研究者使用
零样本组合图像检索(ZS-CIR)是一项具有挑战性的跨模态任务,旨在通过图文联合查询检索目标图像。传统基于反演的方法在训练和推理阶段目标不一致,且输入特征分布存在模态差异。为此,本文提出一个轻量级后处理框架:(1)利用大语言模型将标准图文数据集转换为三元组数据,以文本描述作为目标;(2)设计MoTa-Adapter,一种参数高效微调方法,通过多专家混合层在文本侧引入可学习任务提示,捕获任务先验,并在图像侧调节反演网络输入以匹配下游文本编码器。同时,采用基于熵的优化策略,对难例赋予更高权重。实验表明,该方法在四个主流基准上均达到当前最优性能,显著提升反演类方法表现。所有数据与代码将公开。
原文摘要 · Abstract (English)
As a challenging vision-language task, Zero-Shot Composed Image Retrieval (ZS-CIR) is designed to retrieve target images using bi-modal (image+text) queries. Typical ZS-CIR methods employ an inversion network to generate pseudo-word tokens that effectively represent the input semantics. However, the inversion-based methods suffer from two inherent issues: First, the task discrepancy exists because inversion training and CIR inference involve different objectives. Second, the modality discrepancy arises from the input feature distribution mismatch between training and inference. To this end, we propose a lightweight post-hoc framework, consisting of two components: (1) A new text-anchored triplet construction pipeline leverages a large language model (LLM) to transform a standard image-text dataset into a triplet dataset, where a textual description serves as the target of each triplet. (2) The MoTa-Adapter, a novel parameter-efficient fine-tuning method, adapts the dual encoder to the CIR task using our constructed triplet data. Specifically, on the text side, multiple sets of learnable task prompts are integrated via a Mixture-of-Experts (MoE) layer to capture task-specific priors and handle different types of modifications. On the image side, MoTa-Adapter modulates the inversion network's input to better match the downstream text encoder. In addition, an entropy-based optimization strategy is proposed to assign greater weight to challenging samples, thus ensuring efficient adaptation. Experiments show that, with the incorporation of our proposed components, inversion-based methods achieve significant improvements, reaching state-of-the-art performance across four widely-used benchmarks. All data and code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。