不融合文本特征，让图文查询相互关注，提升多模态检索效果
MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval
- 图文查询交互不直接融合，避免文本主导问题
- 零样本下在4个基准上表现优异，显著提升多模态理解能力
- 适用于需要精准图文匹配的检索场景，如跨模态搜索
当前多模态检索方法通过预训练实现视觉-文本对齐，通常直接融合双模态特征进行交叉参考，但易受文本主导影响，忽略关键视觉信息。本文提出MIRe框架，实现无需特征融合的模态交互：文本查询可关注视觉嵌入，但不将文本信号反馈至视觉表示。同时，构建了基于问答对扩展的预训练数据集用于多模态查询检索。实验表明，该预训练策略显著增强多模态查询理解，在四个多模态检索基准上零样本表现优异。消融实验与分析验证了其有效缓解文本主导问题。代码已开源。
原文摘要 · Abstract (English)
Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。