用多轮对话思路提升多模态模型训练效率与效果
MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model
- 将多轮查询-目标对整合到单次前向传播中，共享上下文表征
- 在500万条数据上训练，检索任务性能超越现有模型
- 适合需要高效多模态对齐的场景，如图文检索、跨模态理解
基于多模态大语言模型（MLLMs）的通用多模态嵌入模型通常采用对比学习对齐不同模态间的查询-目标对表示。然而，传统方法基于“单轮”范式，将每对查询-目标视为独立样本，导致扩展时计算效率低下，且忽略多个查询间可能存在的上下文关联。本文提出多轮对比学习（MuCo），受对话启发，将同一图像对应的多个相关查询-目标对在一次前向传播中处理，共享上下文表征，同步生成多个查询与目标嵌入，显著提升有效批次大小与训练效率。实验基于新构建的500万条多模态多轮数据集（M3T），在MMEB与M-BEIR基准上实现领先检索性能，同时增强跨模态表示一致性。代码与数据集已开源。
原文摘要 · Abstract (English)
Universal Multimodal embedding models built on Multimodal Large Language Models (MLLMs) have traditionally employed contrastive learning, which aligns representations of query-target pairs across different modalities. Yet, despite its empirical success, they are primarily built on a "single-turn" formulation where each query-target pair is treated as an independent data point. This paradigm leads to computational inefficiency when scaling, as it requires a separate forward pass for each pair and overlooks potential contextual relationships between multiple queries that can relate to the same context. In this work, we introduce Multi-Turn Contrastive Learning (MuCo), a dialogue-inspired framework that revisits this process. MuCo leverages the conversational nature of MLLMs to process multiple, related query-target pairs associated with a single image within a single forward pass. This allows us to extract a set of multiple query and target embeddings simultaneously, conditioned on a shared context representation, amplifying the effective batch size and overall training efficiency. Experiments exhibit MuCo with a newly curated 5M multimodal multi-turn dataset (M3T), which yields state-of-the-art retrieval performance on MMEB and M-BEIR benchmarks, while markedly enhancing both training efficiency and representation coherence across modalities. Code and M3T are available at https://github.com/naver-ai/muco
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。