arXiv:2604.12148cs.CV2026-04ACL被引 1

让视频大模型能更好做检索,还能自适应长短视频。

ViLL-E: Video LLM Embeddings for Retrieval

  • 设计新嵌入生成机制,让模型可长思可早停。
  • 在时序定位上提升7%,视频检索最高超双编码器4%。
  • 支持零样本组合检索,适合多任务研究者使用。

视频大语言模型(VideoLLMs)在文本输出任务如视频问答和视频字幕生成中表现优异,但在基于嵌入的检索任务(如文本到视频检索、时刻检索)中性能落后于专用嵌入模型。本文提出ViLL-E(Video-LLM-Embed),一种具备新型嵌入生成机制的统一视频大模型架构,使模型能对复杂视频“思考更久”,对简单视频“提前停止”。通过三阶段训练方法:首先在大规模视频-字幕对上预训练;其次在小规模详细字幕数据集上持续训练;最后在覆盖视频问答、时空定位、视频检索与图文匹配的新型多任务数据集上进行特定任务微调。实验表明,该模型显著提升时序定位性能(平均优于其他VideoLLMs 7%),在视频检索上最高超越双编码器模型4%,达到与顶尖专用嵌入模型相当的水平,同时在视频问答任务上仍保持竞争力。此外,联合对比-生成训练策略解锁了新的零样本能力,在组合视频检索任务上比现有最佳方法高出5%,在长文本检索任务上高出2%。

原文摘要 · Abstract (English)

Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperform specialized embedding-based models in Retrieval tasks, such as Text-toVideo Retrieval and Moment Retrieval. We introduce ViLL-E (Video-LLM-Embed), a unified VideoLLM architecture endowed with a novel embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. We train this model with a three-stage training methodology combining generative and contrastive learning: initial large-scale pre-training with video-caption pairs; followed by continual training on a smaller, detailed-caption dataset; and concluding with task-specific fine-tuning on a novel multi-task dataset covering Video QA, Temporal Localization, Video Retrieval, and Video-Text Matching. Our model significantly improves temporal localization (on avg. 7% over other VideoLLMs) and video retrieval (up to 4% over dual encoder models), achieving performance comparable to state-of-the-art specialized embedding models while remaining competitive on VideoQA tasks. Furthermore, our joint contrastive-generative training unlocks new zero-shot capabilities, significantly outperforming state-of-the-art methods in composed video retrieval (+5% over SotA) and retrieval from long text (+2% over SotA).

视频检索大模型嵌入学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。