arXiv:2511.01588cs.LGcs.CV2025-11被引 4

让多模态大模型生成多个不同视角的嵌入,提升表示能力。

Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization

  • 用可学习前缀控制大模型生成多条并行路径,得到多个嵌入。
  • 通过互信息最小化使各路径差异最大化,同时保持语义一致。
  • 效果显著且推理零额外开销,适合各类模型和分辨率。

嵌入模型是现代人工智能的核心。尽管多模态大语言模型(MLLM)在架构与数据构建上取得进展,但整体范式仍受限于单一输入、单一嵌入、对比监督(SSC),将丰富多样的输入压缩为单一嵌入,未能充分发挥MLLM潜力。本文提出并行解耦框架(PDF),利用MLLM的可控生成能力,通过不同可学习前缀引导共享的MLLM主干生成多条并行路径,从而获得并行嵌入。为促进充分多样性,引入互信息最小化(MIM)作为显式约束,并辅以每路径对比监督以维持语义对齐。双重目标迫使PDF实现稳健的语义覆盖与可泛化嵌入空间。最终,推理时仅需一次前向传播即可访问优异嵌入空间,计算开销可忽略。我们在多个MLLM主干上实例化PDF,验证其在MMEB基准上的有效性。在多种分辨率和模型规模下均取得显著提升:例如,使VLM2Vec-LLaVA-1.6-LR模型性能提升+8.9%(7B),VLM2Vec-Qwen2VL模型分别提升+4.2%(2B)和+3.1%(7B)。效率方面,2B模型仅用一半计算预算即超越基线+2.6%。

原文摘要 · Abstract (English)

Embedding models are a cornerstone of modern AI. Driven by Multimodal Large Language Models (MLLMs), they have made great progress in architecture and data curation, while the holistic paradigm is still limited to SSC, i.e., single input, singular embedding, contrastive supervision, which collapses rich, multifaceted inputs into monolithic embeddings and fails to fully exploit MLLM capabilities. In this paper, we tailor one Parallel Decoupling Framework (PDF) for multimodal embedding learning, by utilizing the proprietary steerability of MLLMs, i.e., their ability to flexibly generate quite differentiated response under explicit instructions. Concretely, PDF conditions a shared MLLM backbone on distinct, learnable prefixes to roll out multiple parallel paths for one input, then relies on these paths to obtain parallel embeddings. To promote full parallel diversity, we employ Mutual Information Minimization (MIM) as an explicit constraint, coupled with per-path contrastive supervision to maintain semantic alignment. Such dual-objectives force PDF to yield robust semantic coverage and a generalizable embedding space. Ultimately, the remarkable embedding space are accessible at inference via one single forward pass, incurring negligible computational overhead. We instantiate PDF on multiple MLLM backbones and prove its effectiveness on MMEB benchmark. Significant gains are consistently achieved across various resolutions and model sizes, e.g., boosting the VLM2Vec-LLaVA-1.6-LR model by a remarkable +8.9% (7B), while the VLM2Vec-Qwen2VL models by +4.2% (2B) and +3.1% (7B). In terms of efficiency, our 2B model surpasses its baseline by +2.6% using only half the computational budget.

多模态嵌入MLLM并行学习互信息

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。