无需训练,通过片段置信度曲线实现视频时间定位的高精度零样本提取。
REZE: Recognition-Based Zero-Shot Extraction for Video Temporal Grounding

- 将视频切片后由VLM评估每段与查询的匹配置信度,外置算法生成时间区间。
- 在QVHighlights上达40.32 mAP,Highlight检测达73.41 HIT@1,刷新无训练方法纪录。
- 适配多类型任务,旧模型用此法可逼近新模型性能,适合快速部署场景。
视频时间定位(VTG)旨在识别自然语言查询对应的视频时间段。传统零样本方法依赖大型视觉语言模型(VLM)直接生成起止时间戳,性能高度依赖模型设计,不同VLM间差异显著。为此,我们提出识别式零样本提取(REZE),一种无需训练的方法:将视频切分为短片段,让模型输出各片段对查询的置信度得分,再通过确定性算法将得分曲线转化为任务所需输出。因时间聚合在模型外完成,REZE可适配单/多区间定位及亮点检测等任务。在QVHighlights上,其单段定位mAP从38.23提升至40.32,亮点检测达44.18 mAP和73.41 HIT@1,创下无训练方法新纪录;其HIT@1甚至超过所有全监督最优方法。我们在三个模型族的七种骨干网络上验证,无论在Charades-STA还是QVHighlights上,均优于直接生成时间戳的方法。此外,发现使用REZE时,早期模型可接近同族新型号的原生性能。
原文摘要 · Abstract (English)
Video temporal grounding (VTG) refers to the task of identifying the time interval in a video that corresponds to a given natural-language query. A common zero-shot strategy asks a large vision-language model (VLM) to generate the start and end timestamps directly, so the result depends heavily on the design and training of the model, and grounding accuracy differs widely from one VLM to another. We therefore propose REcognition-based Zero-shot Extraction (REZE), a simple training-free method that splits the video into short clips, asks the model for a clip-level confidence score for the query, and uses a deterministic algorithm to convert the resulting score curve into the output required by the task. Because temporal aggregation is performed outside the model, REZE adapts to different task outputs, from single- and multi-interval moment retrieval to highlight detection. On QVHighlights, REZE improves the best reported training-free moment-retrieval mAP from 38.23 to 40.32, while on highlight detection it reaches 44.18 mAP and 73.41 HIT@1, establishing a new state of the art among training-free methods. Its HIT@1 also outperforms all fully supervised SoTAs on the QVHighlights test split. We evaluate REZE on seven backbones from three model families. On Charades-STA and QVHighlights, it outperforms direct timestamp generation in every available comparison. We further observe that with REZE an earlier-generation model can approach the native performance of a newer model in its family.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。