arXiv:2411.17454cs.CVcs.CL2024-11

通过特征生成与融合提升跨模态检索性能,解决少样本数据不平衡和特征退化问题。

FLEX-CLIP: Feature-Level GEneration Network Enhanced CLIP for X-shot Cross-modal Retrieval

  • 构建变分自编码器-生成对抗网络,生成伪样本缓解数据不平衡
  • 引入门控残差网络融合特征,降低目标域中特征退化现象
  • 在四个基准数据集上实现7%-15%性能提升,适合少样本跨模态应用

给定一种模态的查询,少样本跨模态检索(CMR)需在另一模态中检索语义相似实例,且目标域类别与源域无交集。相比传统少样本CMR方法,视觉-语言预训练模型如CLIP展现出优异的少样本或零样本学习能力。然而,其仍面临两大挑战:(1)目标域中的特征退化问题;(2)极端数据不平衡。为此,本文提出FLEX-CLIP——一种新型特征级生成网络增强的CLIP方法。FLEX-CLIP包含两个训练阶段:在多模态特征生成阶段,提出复合多模态VAE-GAN网络,捕捉真实特征分布并基于CLIP特征生成伪样本,缓解数据不平衡;在公共空间投影阶段,设计门控残差网络,融合CLIP特征与投影特征,减少在X-shot场景下的特征退化。在四个基准数据集上的实验结果表明,该方法相较当前最优方法提升7%-15%,消融实验证明了对CLIP特征的有效增强。

原文摘要 · Abstract (English)

Given a query from one modality, few-shot cross-modal retrieval (CMR) retrieves semantically similar instances in another modality with the target domain including classes that are disjoint from the source domain. Compared with classical few-shot CMR methods, vision-language pretraining methods like CLIP have shown great few-shot or zero-shot learning performance. However, they still suffer challenges due to (1) the feature degradation encountered in the target domain and (2) the extreme data imbalance. To tackle these issues, we propose FLEX-CLIP, a novel Feature-level Generation Network Enhanced CLIP. FLEX-CLIP includes two training stages. In multimodal feature generation, we propose a composite multimodal VAE-GAN network to capture real feature distribution patterns and generate pseudo samples based on CLIP features, addressing data imbalance. For common space projection, we develop a gate residual network to fuse CLIP features with projected features, reducing feature degradation in X-shot scenarios. Experimental results on four benchmark datasets show a 7%-15% improvement over state-of-the-art methods, with ablation studies demonstrating enhancement of CLIP features.

跨模态检索少样本学习特征生成CLIP增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。