构建首个精细时序视频重检索基准,提升体育视频精准查找能力。
From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos
- 用大语言模型生成跨视频动作差异提示,构造多目标匹配对
- 提出两阶段框架,在零样本下mAP@50达7.51,微调后达27.22
- 专为体操跳水等快速动作设计,适合视频编辑与精彩片段生成
组成式视频检索(CoVR)旨在给定查询视频和描述意图修改的文本时,检索出目标视频。现有CoVR基准主要关注外观变化或粗粒度事件变更,无法评估捕捉细微、快速时序差异的能力。本文提出首个大规模精细时序CoVR基准TF-CoVR,聚焦体操与跳水场景,基于FineGym与FineDiving数据集构建18万组三元组。与以往仅关联同一视频内单一目标段不同,本工作通过LLM生成不同视频间动作标签差异,构造每对<查询, 修改>关联3.9个有效目标视频,更贴近真实场景如体育精彩集锦生成。为此提出TF-CoVR-Base框架:先在细粒度动作分类任务上预训练视频编码器以获取时序判别嵌入;再采用对比学习对齐组合查询与候选视频。首次全面评估图像、视频及通用多模态嵌入(GME)模型在零样本与微调下的表现。在TF-CoVR上,零样本下mAP@50从LanguageBind的5.92提升至7.51,微调后将最佳结果从19.83提高至27.22。
原文摘要 · Abstract (English)
Composed Video Retrieval (CoVR) retrieves a target video given a query video and a modification text describing the intended change. Existing CoVR benchmarks emphasize appearance shifts or coarse event changes and therefore do not test the ability to capture subtle, fast-paced temporal differences. We introduce TF-CoVR, the first large-scale benchmark dedicated to temporally fine-grained CoVR. TF-CoVR focuses on gymnastics and diving, and provides 180K triplets drawn from FineGym and FineDiving datasets. Previous CoVR benchmarks, focusing on temporal aspect, link each query to a single target segment taken from the same video, limiting practical usefulness. In TF-CoVR, we instead construct each <query, modification> pair by prompting an LLM with the label differences between clips drawn from different videos; every pair is thus associated with multiple valid target videos (3.9 on average), reflecting real-world tasks such as sports-highlight generation. To model these temporal dynamics, we propose TF-CoVR-Base, a concise two-stage training framework: (i) pre-train a video encoder on fine-grained action classification to obtain temporally discriminative embeddings; (ii) align the composed query with candidate videos using contrastive learning. We conduct the first comprehensive study of image, video, and general multimodal embedding (GME) models on temporally fine-grained composed retrieval in both zero-shot and fine-tuning regimes. On TF-CoVR, TF-CoVR-Base improves zero-shot mAP@50 from 5.92 (LanguageBind) to 7.51, and after fine-tuning raises the state-of-the-art from 19.83 to 27.22.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。