让多模态嵌入模型智能选择是否推理,提升效果同时降低延迟
MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
- 将推理设为可调节的隐变量,按需选择是否启用
- 在MMEB-V2上以40亿参数达到71.2分新纪录
- 适合需要高效高精度嵌入的应用场景
大语言模型在多模态嵌入任务中表现优异,但其生成式推理能力仍未被充分利用。直接将思维链推理融入嵌入学习会带来两个根本问题:一是实例级推理与成对对比监督之间存在结构错位,导致模型仅学习推理形式而非实质内容;二是推理并非对所有输入都有益,强制执行会增加计算开销并掩盖简单样本的语义特征。为此,我们提出MMEmb-R1,一种自适应推理增强的多模态嵌入框架。将推理视为隐变量,引入基于反事实干预的成对感知推理选择机制,识别对查询-目标对齐有益的推理路径。同时采用强化学习策略,仅在必要时调用推理。在MMEB-V2基准上的实验表明,该模型仅用40亿参数即取得71.2分,创下新纪录,显著降低推理开销与推理延迟。
原文摘要 · Abstract (English)
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。