无需训练即可精准定位长视频中的关键片段,效率远超现有方法。
Point to Span: Zero-Shot Moment Retrieval for Navigating Unseen Hour-Long Videos
- 通过自适应生成候选片段,避免搜索阶段的候选爆炸
- 利用查询分解实现高效精炼,不依赖昂贵视觉语言模型
- 首个无需训练就能处理小时级视频的零样本定位框架
零样本长视频片段检索(ZLVMR)是指在不进行特定任务训练的前提下,仅通过自然语言查询定位长达一小时视频中的时间片段。该任务的核心挑战在于无法对整段长视频一次性处理。现有方法普遍采用‘搜索-精炼’范式:先快速缩小候选范围,再对局部内容精细分析。然而,传统监督学习存在扩展性差、泛化能力弱的问题;而现有零样本方法也面临双重困境:(1) 搜索阶段的启发式策略导致候选片段爆炸;(2) 精炼阶段易受语义偏差影响,需依赖高成本视觉语言模型验证,带来巨大计算开销。本文提出无需训练的Point-to-Span(P2S)框架,通过两个核心创新突破瓶颈:一是‘自适应片段生成器’防止搜索阶段候选爆炸,二是‘查询分解’实现无需高成本VLM验证的精炼机制。据我们所知,P2S是首个可直接用于小时级视频零样本定位的框架,在MAD数据集上[email protected]指标领先最先进监督方法3.7个百分点。
原文摘要 · Abstract (English)
Zero-shot Long Video Moment Retrieval (ZLVMR) is the task of identifying temporal segments in hour-long videos using a natural language query without task-specific training. The core technical challenge of LVMR stems from the computational infeasibility of processing entire lengthy videos in a single pass. This limitation has established a 'Search-then-Refine' approach, where candidates are rapidly narrowed down, and only those portions are analyzed, as the dominant paradigm for LVMR. However, existing approaches to this paradigm face severe limitations. Conventional supervised learning suffers from limited scalability and poor generalization, despite substantial resource consumption. Yet, existing zero-shot methods also fail, facing a dual challenge: (1) their heuristic strategies cause a 'search' phase candidate explosion, and (2) the 'refine' phase, which is vulnerable to semantic discrepancy, requires high-cost VLMs for verification, incurring significant computational overhead. We propose \textbf{P}oint-\textbf{to}-\textbf{S}pan (P2S), a novel training-free framework to overcome this challenge of inefficient 'search' and costly 'refine' phases. P2S overcomes these challenges with two key innovations: an 'Adaptive Span Generator' to prevent the search phase candidate explosion, and 'Query Decomposition' to refine candidates without relying on high-cost VLM verification. To our knowledge, P2S is the first zero-shot framework capable of temporal grounding in hour-long videos, outperforming supervised state-of-the-art methods by a significant margin (e.g., +3.7\% on [email protected] on MAD).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。