arXiv:2511.22141cs.CL2025-11中稿 · PACLIC2025被引 1

用伪正样本标准化相似度,解决图文检索中的模态差异问题。

Bridging the Modality Gap by Similarity Standardization with Pseudo-Positive Samples

  • 通过构建伪正样本计算模态统计量,统一跨模态相似度尺度。
  • 在不同模态间检索时,MMQA上召回率提升64%,WebQA上提升28%。
  • 无需人工标注,适用于多种视觉语言模型,尤其适合跨模态任务。

视觉-语言模型(VLMs)虽已实现有效的跨模态检索,但当文本与图像共存于数据库时,模态间的相似度得分存在尺度差异,即“模态间隙”,影响检索精度。现有方法多依赖人工标注数据进行微调。本文提出一种基于伪数据构造的相似度标准化方法:首先计算查询与其配对数据在文本或图像模态下的相似度均值与方差,利用这些模态特定统计量对所有相似度进行标准化,使跨模态比较在同一尺度上进行。伪正样本由与查询具有最高余弦相似度的文本和图像候选者构成。我们在七个VLM上使用两个多模态问答基准(MMQA和WebQA)进行评估,每个问题需检索文本或图像数据。实验结果表明,该方法显著提升检索性能,在不同模态间检索时,MMQA上平均Recall@20提升64%,WebQA上提升28%。相比E5-V通过图像描述缓解模态间隙的方法,本方法更有效弥合模态差距。

原文摘要 · Abstract (English)

Advances in vision-language models (VLMs) have enabled effective cross-modality retrieval. However, when both text and images exist in the database, similarity scores would differ in scale by modality. This phenomenon, known as the modality gap, hinders accurate retrieval. Most existing studies address this issue with manually labeled data, e.g., by fine-tuning VLMs on them. In this work, we propose a similarity standardization approach with pseudo data construction. We first compute the mean and variance of the similarity scores between each query and its paired data in text or image modality. Using these modality-specific statistics, we standardize all similarity scores to compare on a common scale across modalities. These statistics are calculated from pseudo pairs, which are constructed by retrieving the text and image candidates with the highest cosine similarity to each query. We evaluate our method across seven VLMs using two multi-modal QA benchmarks (MMQA and WebQA), where each question requires retrieving either text or image data. Our experimental results show that our method significantly improves retrieval performance, achieving average Recall@20 gains of 64% on MMQA and 28% on WebQA when the query and the target data belong to different modalities. Compared to E5-V, which addresses the modality gap through image captioning, we confirm that our method more effectively bridges the modality gap.

跨模态检索模态对齐相似度标准化伪数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。