让多模态大模型先推理再生成嵌入,提升检索效果
Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval
- 用推理过程引导嵌入生成,增强上下文感知能力
- 在MMEB基准上检索准确率提升4.9%(相比无推理基线)
- 适合需要高精度多模态检索的应用场景
多模态嵌入广泛用于多模态检索等下游任务,实现跨模态在共享表示空间中的对齐。尽管近期研究表明多模态大语言模型(MLLMs)可作为强大的嵌入提取器,但现有方法将嵌入提取视为直接编码步骤,忽略了MLLM具备的推理生成能力。本文提出推理引导嵌入(RGE),保留MLLM的生成式推理过程,并与对比学习相结合。该方法首先根据指令进行结构化推理生成,然后在推理完成后提取表示。这种设计增强了嵌入中的上下文条件推理信号,从而提升多模态表征质量。在MMEB基准上的实验表明,推理引导条件使多模态检索性能相比非推理基线提升4.9%,证实显式推理能有效增强嵌入质量。
原文摘要 · Abstract (English)
Multimodal embeddings are widely used in downstream tasks such as multimodal retrieval, enabling alignment of interleaved modalities in a shared representation space. While recent studies show that Multimodal Large Language Models (MLLMs) can serve as strong embedding extractors, existing approaches treat embedding extraction as a direct encoding step, overlooking the fact that MLLMs possess the generative capability for reasoning that could be leveraged to enhance representation quality. In this work, we explore how to explicitly incorporate reasoning into the embedding process. To this end, we propose Reasoning Guided Embeddings (RGE), which preserves the generative rationale process of MLLMs and couples it with contrastive training. Our method first enables the model to perform structured rationale generation conditioned on the instruction, and then extracts representations after reasoning has unfolded. This simple design enhances the context-conditional inference signals within the embedding, leading to improved multimodal representation quality. Experiments on the MMEB benchmark show that reasoning-guided conditioning improves multimodal retrieval performance by 4.9% over the non-reasoning baseline, confirming that explicit reasoning can effectively enhance embedding quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。