用统一模型实现视频全文与片段检索,支持复杂多模态查询。
VeRVE: Versatile Retrieval for Videos via Unified Embeddings
- 基于共享大模型提取视觉与文本嵌入,通过对比对齐提升检索效率。
- 在零样本视频检索上超越其他多模态大模型方法,达领先水平。
- 无需再训练即可实现片段定位,适合需要灵活查询的场景。
现代视频检索系统需应对从文档级检索、细粒度时刻定位到灵活多模态查询等多样化任务。专用架构虽在大规模数据上训练特定编码器,表现优异,但难以处理复合多模态查询;而基于多模态大语言模型(MLLM)的方法虽支持丰富查询,检索性能仍显著低于专用系统。本文提出VeRVE,一个基于MLLM的通用视频检索框架,集成文档级与时刻级检索能力,并在单一架构中支持复合多模态查询。利用共享MLLM主干生成的视觉与文本嵌入进行对比对齐,实现高效基于嵌入的候选搜索。该嵌入模型在70万对图文数据样本上采用低秩适配(LoRA)高效训练,在零样本视频检索任务中优于其他基于MLLM的方法。此外,相同模型无需进一步训练即可在零样本时刻检索上取得竞争力结果,并在零样本复合视频检索上达到最先进水平。通过额外训练对嵌入搜索选出的候选进行重排序,模型性能显著超越现有基于MLLM的检索系统,接近最先进的专用模型水平。
原文摘要 · Abstract (English)
Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval, fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VeRVE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。