统一嵌入与排序,让多模态检索更精准。
UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
- 用配对判别推理替代独立分析,显式捕捉正负样本差异
- 在MMEB-V2上达到最新最佳性能,支持可调预算推理
- 适合需要细粒度判断的多模态检索任务
通用多模态检索旨在支持多样化的指令感知检索任务,要求兼具高效的全库匹配与精细的语义推理能力。现有基于多模态大模型(MLLM)的嵌入方法通常从隐藏状态提取表示,而思维链(CoT)推理通过将中间语义证据编码至表示空间,正成为嵌入增强的有力策略。然而,现有CoT方法通常对查询与候选项进行孤立的逐项推理,无法提供明确证据以区分正样本与语义混淆的困难负样本。此外,对比嵌入虽能捕捉全局相似性,但在需答案验证、类别判断或细粒度推理的元任务中表现不佳。本文提出UMER——一种统一多模态嵌入与排序的框架。UMER以配对感知判别推理取代逐项反思,通过比较查询-候选对来识别指令相关匹配与差异证据。UMER在单一MLLM中联合学习对比嵌入以实现高效全局匹配,以及判别排序以进行显式的成对相关性判断。进一步采用互补的相互蒸馏策略,将在嵌入与排序函数间可靠的成对偏好进行迁移。在MMEB-V2基准测试中,UMER在可比实验设置下取得当前最优性能,并支持可调节推理预算。
原文摘要 · Abstract (English)
Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-grained semantic reasoning. Recent MLLM-based embedding methods typically derive representations from hidden states, while Chain-of-Thought (CoT) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with Pair-Aware Discriminative Reasoning, which compares query--candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On the MMEB-V2 benchmark, UMER achieves state-of-the-art performance under comparable experimental settings while supporting budget-adjustable inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。