不需微调，用文本对齐让MLLM直接实现视频图文检索突破
VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
- 利用MLLM中间层嵌入+校准头，零样本实现强检索能力
- 在多个基准上超越现有方法，达到最新最优性能
- 仅用文本对齐无需视觉监督，适合资源受限场景
近期研究将生成式多模态大模型（MLLM）用于视觉任务的嵌入提取，通常通过微调生成通用表示，但在视频任务上的表现仍逊于视频基础模型（VFMs）。本文聚焦于利用MLLM进行视频-文本嵌入与检索。我们首先进行系统性逐层分析，发现预训练的MLLM中间层已编码大量任务相关信息。基于此，我们证明结合中间层嵌入与校准后的MLLM头部可实现强大的零样本检索性能，无需任何训练。在此基础上，提出一种轻量级文本对齐策略，将密集视频字幕映射为简短摘要，实现无需视觉监督的任务相关视频-文本嵌入学习。令人惊喜的是，仅在文本层面调整，无需额外微调，本方法在多个常见视频检索基准上显著优于现有方法，达到当前最佳水平。
原文摘要 · Abstract (English)
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。