arXiv:2605.14448cs.CVcs.CL2026-05

让模型只在必要时思考,提升多模态嵌入效率与精度

Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture

论文配图:Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture
图 1 · 摘自论文原文
  • 采用双LoRA架构,共享冻结主干网络,减少参数开销
  • 自适应路由机制按需生成思维链,减少50%推理令牌消耗
  • 适合追求高效高精度多模态检索的开发者和研究者

多模态大语言模型(MLLMs)已成为多模态嵌入的强大基础。现有方法将思维链(CoT)引入嵌入流程以提升检索质量,但存在模型规模大、推理成本高的问题。通常采用独立的推理器与嵌入器,参数开销大,且对所有输入均无差别生成CoT。我们观察到:简单输入下,判别性嵌入已表现良好,冗余推理甚至会误导模型,降低性能。为此,提出Think When Needed(TWN),一种具备自适应推理能力的统一多模态嵌入框架。TWN采用双LoRA架构,在共享冻结主干上附加推理与嵌入适配器,通过接口断开梯度以缓解联合优化带来的梯度冲突,同时保持参数接近单模型水平。在此基础上,设计自监督路由门实现按输入动态决策是否生成CoT,跳过不必要的推理以降低推理开销,并可能提升检索质量。进一步探索嵌入引导的强化学习以优化CoT质量,超越监督训练。在MMEB-V2的78个任务上,TWN达到当前最优嵌入质量,相比现有生成式方法显著更高效:仅需主干模型3-5%额外参数,且推理令牌数最多减少50%。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) have emerged as a powerful backbone for multimodal embeddings. Recent methods introduce chain-of-thought (CoT) reasoning into the embedding pipeline to improve retrieval quality, but remain costly in both model size and inference cost. They typically employ separate reasoner and embedder with substantial parameter overhead, and generate CoT indiscriminately for every input. However, we observe that for simple inputs, discriminative embeddings already perform well, and redundant reasoning can even mislead the model, degrading performance. To address these limitations, we propose Think When Needed (TWN), a unified multimodal embedding framework with adaptive reasoning. TWN introduces a dual-LoRA architecture that attaches reasoning and embedding adapters to a shared frozen backbone, detaching gradients at their interface to mitigate gradient conflicts introduced by joint optimization while keeping parameters close to a single model. Building on this, an adaptive think mechanism uses a self-supervised routing gate to decide per input whether to generate CoT, skipping unnecessary reasoning to reduce inference overhead and even improve retrieval quality. We further explore embedding-guided RL to optimize CoT quality beyond supervised training. On the 78 tasks of MMEB-V2, TWN achieves state-of-the-art embedding quality while being substantially more efficient than existing generative methods, requiring only 3-5% additional parameters relative to the backbone and up to 50% fewer reasoning tokens compared to the full generative mode.

多模态嵌入自适应推理LoRA高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。