解决图文混合搜索中嵌入空间的模态差距问题
Closing the Modality Gap for Mixed Modality Search
- 提出轻量级后处理校准方法GR-CLIP，消除图像与文本嵌入的分离
- 在MixBench上使NDCG@10提升26个百分点，优于先进模型
- 仅需原模型75分之一计算量，适合实际部署
混合模态搜索——从包含图像、文本和多模态文档的异构语料库中检索信息——是重要但研究不足的实际应用。本文分析发现，对比视觉-语言模型（如CLIP）在该任务中存在显著模态差距：图像与文本嵌入在嵌入空间中形成分离聚类，导致模态内排序偏差与跨模态融合失败。为此，我们提出GR-CLIP，一种轻量级后处理校准方法，可有效消除CLIP嵌入空间中的模态差距。在首个专为混合模态搜索设计的基准测试MixBench上，GR-CLIP相较CLIP提升NDCG@10达26个百分点，超越近期视觉-语言生成嵌入模型4个百分点，同时仅需75倍更少的计算资源。
原文摘要 · Abstract (English)
Mixed modality search -- retrieving information across a heterogeneous corpus composed of images, texts, and multimodal documents -- is an important yet underexplored real-world application. In this work, we investigate how contrastive vision-language models, such as CLIP, perform on the mixed modality search task. Our analysis reveals a critical limitation: these models exhibit a pronounced modality gap in the embedding space, where image and text embeddings form distinct clusters, leading to intra-modal ranking bias and inter-modal fusion failure. To address this issue, we propose GR-CLIP, a lightweight post-hoc calibration method that removes the modality gap in CLIP's embedding space. Evaluated on MixBench -- the first benchmark specifically designed for mixed modality search -- GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP, surpasses recent vision-language generative embedding models by 4 percentage points, while using 75x less compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。