用纯文本训练让多模态大模型更好理解视频搜索中的细微差别。
Adapting MLLMs for Nuanced Video Retrieval
- 将文本生成模型转为嵌入模型，用对比损失训练
- 在多个基准上达到顶尖性能，尤其擅长时间、否定和复合查询
- 纯文本训练减少模态差距，提升嵌入空间组织性
本文旨在构建一个能捕捉搜索查询与候选视频之间细微关系的嵌入模型。关注三类细微检索：(i) 时间性，如区分“开门”与“关门”这类相反动作；(ii) 否定性，处理含“不”“无”等否定词的查询；(iii) 多模态复合性，即查询包含视频与文本编辑指令。目标是开发统一嵌入模型有效应对这些细微差别。为此，我们将一个预训练的多模态大语言模型（MLLM）从文本生成任务转为嵌入模型，并仅用文本数据进行微调，采用精心采样的难例负样本和对比损失。尽管训练仅使用文本，该方法在所有细微视频检索基准上均取得当前最优性能。进一步分析表明，这种提升源于文本训练降低了文本与视频嵌入间的模态差距，使嵌入空间组织更合理。
原文摘要 · Abstract (English)
Our objective is to build an embedding model that captures the nuanced relationship between a search query and candidate videos. We cover three aspects of nuanced retrieval: (i) temporal, (ii) negation, and (iii) multimodal. For temporal nuance, we consider chiral actions that need distinguishing between temporally opposite actions like "opening a door" vs. "closing a door". For negation, we consider queries with negators such as "not", "none" that allow user to specify what they do not want. For multimodal nuance, we consider the task of composed retrieval where the query comprises a video along with a text edit instruction. The goal is to develop a unified embedding model that handles such nuances effectively. To that end, we repurpose a Multimodal Large Language Model (MLLM) trained to generate text into an embedding model. We fine-tune it with a contrastive loss on text alone with carefully sampled hard negatives that instill the desired nuances in the learned embedding space. Despite the text-only training, our method achieves state of the art performance on all benchmarks for nuanced video retrieval. We also analyze how this improvement is achieved, and show that text-only training reduces the modality gap between text and video embeddings leading to better organization of the embedding space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。