用生成式模型提升多模态检索效果,让模型自己判断相关性。
ReMatch: Boosting Representation through Matching for Multimodal Retrieval
- 用生成式推理让模型自回归判断查询与文档相关性
- 在MMEB上达到新SOTA,零样本泛化能力显著
- 适合追求高精度检索和跨数据集迁移的开发者
我们提出ReMatch,一种利用多模态大语言模型(MLLM)生成能力进行多模态检索的框架。以往方法将MLLM仅当作编码器,忽视其生成特性与组合推理能力。ReMatch通过端到端训练,引入对话式生成匹配阶段,利用同一MLLM对原始输入及自身投影嵌入进行自回归相关性判断,提供实例级判别监督,增强难负样本梯度,并保留原模型的组合优势。为获得更丰富的语义嵌入,采用多个可学习标记增强输入,生成细粒度、相互正交且推理开销低的嵌入。基于高性能基线,整合上述设计形成高效训练方案,在大规模多模态嵌入基准(MMEB)上取得新SOTA。实验显示其在五个数据集上零样本泛化表现优异,体现强鲁棒性与可迁移性。
原文摘要 · Abstract (English)
We present ReMatch, a framework that leverages the generative strength of MLLMs for multimodal retrieval. Previous approaches treated an MLLM as a simple encoder, ignoring its generative nature, and under-utilising its compositional reasoning and world knowledge. We instead train the embedding MLLM end-to-end with a chat-style generative matching stage. The matching stage uses the same MLLM to autoregressively decide relevance from multi-view inputs, including both raw data and its own projected embeddings for each query and document. It provides instance-wise discrimination supervision that complements a standard contrastive loss, offering stronger gradients on hard negatives and preserving the compositional strengths of the original MLLM. To obtain semantically richer multimodal embeddings, we use multiple learnable tokens to augment each input, generating fine-grained contextual, mutually orthogonal embeddings with low inference cost. Leveraging our established high-performance baseline,we assemble the ideas mentioned above into a powerful training recipe and achieve a new state-of-the-art on the Massive Multimodal Embedding Benchmark (MMEB). Our experiments show particularly strong zero-shot generalization results on five datasets, highlighting the robustness and transferability of ReMatch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。