arXiv:2601.12193cs.CV2026-01被引 2

用统一模型实现视频全文与片段检索,支持复杂多模态查询。

VeRVE: Versatile Retrieval for Videos via Unified Embeddings

  • 基于共享大模型提取视觉与文本嵌入,通过对比对齐提升检索效率。
  • 在零样本视频检索上超越其他多模态大模型方法,达领先水平。
  • 无需再训练即可实现片段定位,适合需要灵活查询的场景。

现代视频检索系统需应对从文档级检索、细粒度时刻定位到灵活多模态查询等多样化任务。专用架构虽在大规模数据上训练特定编码器,表现优异,但难以处理复合多模态查询;而基于多模态大语言模型(MLLM)的方法虽支持丰富查询,检索性能仍显著低于专用系统。本文提出VeRVE,一个基于MLLM的通用视频检索框架,集成文档级与时刻级检索能力,并在单一架构中支持复合多模态查询。利用共享MLLM主干生成的视觉与文本嵌入进行对比对齐,实现高效基于嵌入的候选搜索。该嵌入模型在70万对图文数据样本上采用低秩适配(LoRA)高效训练,在零样本视频检索任务中优于其他基于MLLM的方法。此外,相同模型无需进一步训练即可在零样本时刻检索上取得竞争力结果,并在零样本复合视频检索上达到最先进水平。通过额外训练对嵌入搜索选出的候选进行重排序,模型性能显著超越现有基于MLLM的检索系统,接近最先进的专用模型水平。

原文摘要 · Abstract (English)

Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval, fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VeRVE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models.

视频检索多模态大模型嵌入对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。