让AI像人一样分步搜索，精准定位长视频中的事件时间点
ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos
- 采用递归机制，从大范围逐步聚焦到精确时间点
- 在MAD数据集上比现有方法高2.6%的定位准确率
- 适合需要精确定位长视频内容的研究与应用
大型语言模型（LLMs）在长文本信息检索中表现优异，但其视觉语言模型（VLMs）在处理时长一小时以上的视频时，尤其在时间定位任务上面临挑战。主要原因在于帧数限制导致关键时间细节丢失。为此，我们提出ReVisionLLM，一种递归视觉语言模型，用于在超长视频中定位事件。受人类搜索策略启发，模型先定位宽泛的兴趣段，再逐步修正焦点以精确识别事件起止时间。该模型可无缝处理从分钟到小时级的视频。我们设计了一种分层训练策略：从短片段开始捕捉显著事件，逐步扩展至更长视频。据我们所知，ReVisionLLM是首个实现小时级视频时间定位的VLM，在多个数据集上显著优于现有最先进方法，于MAD数据集上达+2.6% [email protected]。代码已开源：https://github.com/Tanveer81/ReVisionLLM。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at retrieving information from lengthy text, but their vision-language counterparts (VLMs) face difficulties with hour-long videos, especially for temporal grounding. Specifically, these VLMs are constrained by frame limitations, often losing essential temporal details needed for accurate event localization in extended video content. We propose ReVisionLLM, a recursive vision-language model designed to locate events in hour-long videos. Inspired by human search strategies, our model initially targets broad segments of interest, progressively revising its focus to pinpoint exact temporal boundaries. Our model can seamlessly handle videos of vastly different lengths, from minutes to hours. We also introduce a hierarchical training strategy that starts with short clips to capture distinct events and progressively extends to longer videos. To our knowledge, ReVisionLLM is the first VLM capable of temporal grounding in hour-long videos, outperforming previous state-of-the-art methods across multiple datasets by a significant margin (+2.6% [email protected] on MAD). The code is available at https://github.com/Tanveer81/ReVisionLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。