用真实视频无监督预训练，让模型学会定位视频中的语言描述片段。
Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild
- 用5万段真实视频和20万条伪标注做无监督预训练。
- 提出双阶段修正算法，使模型在零样本下达到监督模型75%以上性能。
- 适合研究视频理解、少样本学习或降低标注成本的开发者。
给定自然语言查询，视频片段定位旨在定位未剪辑视频中描述的时间片段。该任务的主要挑战是严重依赖人工标注进行训练。与现有工作直接在人工标注数据上训练模型不同，本文提出一种新范式：在无标注的真实世界视频上预训练模型。为此，我们引入视频片段定位预训练（Vid-Morp），一个大规模数据集，仅需少量人工干预，包含超过50,000段野外采集的视频和200,000条伪标注。然而，在这些不完美伪标注上直接预训练面临重大挑战，包括语句-视频对不匹配和时间边界不准确。为此，我们提出ReCorrect算法，包含两个阶段：语义引导修正与记忆共识校正。语义引导修正通过利用语义相似性清洗无关数据并初步调整时间边界；记忆共识校正阶段通过记忆库追踪模型预测，基于记忆内一致性逐步修正时间边界。大量实验表明，ReCorrect在多个下游任务中具有强大泛化能力。零样本设置下，ReCorrect在两个基准上分别达到最佳全监督性能的75%和80%；无监督设置下，性能接近85%。代码、数据集和预训练模型已公开于https://github.com/baopj/Vid-Morp。
原文摘要 · Abstract (English)
Given a natural language query, video moment retrieval aims to localize the described temporal moment in an untrimmed video. A major challenge of this task is its heavy dependence on labor-intensive annotations for training. Unlike existing works that directly train models on manually curated data, we propose a novel paradigm to reduce annotation costs: pretraining the model on unlabeled, real-world videos. To support this, we introduce Video Moment Retrieval Pretraining (Vid-Morp), a large-scale dataset collected with minimal human intervention, consisting of over 50K videos captured in the wild and 200K pseudo annotations. Direct pretraining on these imperfect pseudo annotations, however, presents significant challenges, including mismatched sentence-video pairs and imprecise temporal boundaries. To address these issues, we propose the ReCorrect algorithm, which comprises two main phases: semantics-guided refinement and memory-consensus correction. The semantics-guided refinement enhances the pseudo labels by leveraging semantic similarity with video frames to clean out unpaired data and make initial adjustments to temporal boundaries. In the following memory-consensus correction phase, a memory bank tracks the model predictions, progressively correcting the temporal boundaries based on consensus within the memory. Comprehensive experiments demonstrate ReCorrect's strong generalization abilities across multiple downstream settings. Zero-shot ReCorrect achieves over 75% and 80% of the best fully-supervised performance on two benchmarks, while unsupervised ReCorrect reaches about 85% on both. The code, dataset, and pretrained models are available at https://github.com/baopj/Vid-Morp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。