无需微调视觉语言模型，高效实现视频检索的语义理解。
PREGEN: Uncovering Latent Thoughts in Composed Video Retrieval
- 用冻结的预训练模型提取视频与文本的深层特征，搭配轻量编码器。
- 在标准数据集上召回率提升超27%，复杂文本修改下仍表现优异。
- 适合追求高效、低资源、强泛化能力的视频检索研究者。
组合视频检索（CoVR）旨在根据查询视频和修改文本检索目标视频。现有方法未能充分挖掘现代视觉-语言模型（VLM）的能力，或使用过时架构，或需高成本微调与慢速生成。我们提出PREGEN（PRE GENeration extraction），一种高效且强大的CoVR框架，通过将冻结的预训练VLM与轻量编码模型结合，完全避免VLM微调。将查询视频与修改文本输入VLM后，提取每层最后标记的隐藏状态，再对这些聚合表示进行简单编码，生成语义丰富且紧凑的嵌入用于检索。PREGEN显著超越以往方法，在标准CoVR基准上实现+27.23和+69.59的Recall@1提升。该方法在不同VLM主干上均表现出鲁棒性，并对更复杂的文本修改展现出强大零样本泛化能力，验证其有效性与语义建模能力。
原文摘要 · Abstract (English)
Composed Video Retrieval (CoVR) aims to retrieve a video based on a query video and a modifying text. Current CoVR methods fail to fully exploit modern Vision-Language Models (VLMs), either using outdated architectures or requiring computationally expensive fine-tuning and slow caption generation. We introduce PREGEN (PRE GENeration extraction), an efficient and powerful CoVR framework that overcomes these limitations. Our approach uniquely pairs a frozen, pre-trained VLM with a lightweight encoding model, eliminating the need for any VLM fine-tuning. We feed the query video and modifying text into the VLM and extract the hidden state of the final token from each layer. A simple encoder is then trained on these pooled representations, creating a semantically rich and compact embedding for retrieval. PREGEN significantly advances the state of the art, surpassing all prior methods on standard CoVR benchmarks with substantial gains in Recall@1 of +27.23 and +69.59. Our method demonstrates robustness across different VLM backbones and exhibits strong zero-shot generalization to more complex textual modifications, highlighting its effectiveness and semantic capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。