arXiv:2511.00405cs.LGcs.AI2025-11中稿 · ICLR被引 32

提出生成式多模态嵌入框架,让模型通过推理生成更优嵌入。

UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings

  • 两阶段训练:先微调赋予推理能力,再用强化学习优化生成质量
  • 生成嵌入在78项任务上显著优于传统判别式嵌入
  • 适合追求可解释性与推理能力的多模态研究者

多模态大语言模型(MLLMs)的成功推动了多模态嵌入的发展,但现有模型仍以判别式为主,难以发挥推理生成范式的潜力。本文提出UMe-R1,一个统一生成范式的通用多模态嵌入框架,采用两阶段训练策略:冷启动监督微调赋予模型推理能力,使其生成判别式与生成式嵌入;后续强化学习进一步提升推理能力并优化生成嵌入质量。本工作揭示四大关键发现:1)利用MLLM的强大生成推理能力,生成嵌入显著优于传统判别式嵌入;2)判别式与生成式嵌入互补,联合表现远超单一模式;3)强化学习能有效优化生成嵌入,建立可扩展优化范式;4)推理时重复采样显著提升下游任务覆盖率(pass@k),展现生成嵌入的推理扩展潜力。在涵盖视频、图像与视觉文档的78项任务的MMEB-V2基准上,UMe-R1显著超越传统判别式嵌入模型,为更具可解释性与推理驱动的生成式多模态嵌入奠定基础。代码、模型与数据集将公开于https://github.com/XMUDeepLIT/UME-R1。

原文摘要 · Abstract (English)

The remarkable success of multimodal large language models (MLLMs) has driven advances in multimodal embeddings, yet existing models remain inherently discriminative, limiting their ability to benefit from reasoning-driven generation paradigm. In this work, we pioneer the exploration of generative embeddings, unifying embedding tasks within a generative paradigm. We propose UME-R1, a universal multimodal embedding framework consisting of a two-stage training strategy: a cold-start supervised fine-tuning equips the model with reasoning capabilities and enables it to generate both discriminative and generative embeddings; a subsequent reinforcement learning enhances reasoning and further optimizes generative embedding quality. This pioneering work reveals four key insights: 1) generative embeddings unlock substantial performance gains over conventional discriminative embeddings by leveraging the powerful generative reasoning capabilities of MLLMs; 2) discriminative and generative embeddings are complementary, whose combined oracle performance far exceeding that of either alone; 3) RL can effectively enhance generative embeddings, establishing a scalable optimization paradigm.; 4) repeated sampling at inference boosts downstream task coverage (pass@k), highlighting the inference-time scalability potential of generative embeddings. Evaluated on the MMEB-V2 benchmark across 78 tasks spanning video, image, and visual documents, UME-R1 significantly outperforms conventional discriminative embedding models and offers a foundation for more interpretable, reasoning-driven generative multimodal embeddings. Our code, models, and datasets will be publicly available at https://github.com/XMUDeepLIT/UME-R1.

多模态嵌入生成式推理MLLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。