arXiv:2602.23294cs.CV2026-02中稿 · ECCV

解决长视频时空定位难题,实现高效精准的长时序目标定位。

Towards Long-Form Spatio-Temporal Video Grounding

  • 采用自回归架构逐帧处理长视频,避免一次性处理全部帧的负担。
  • 引入空间与时间记忆库,结合选择策略提升上下文相关性,性能显著提升。
  • 提出级联式时空定位设计,用精细空间信息辅助复杂时间定位,适合长视频场景。

在真实场景中,视频时长可达数分钟甚至数小时。然而,现有时空视频定位(STVG)研究主要针对几十秒以内的短视频,通常不足一分钟,限制了实际应用。本文探索长时序视频定位(LF-STVG),旨在定位长视频中的目标。相比短视频,长视频具有更长的时间跨度和更多无关信息,使传统一次性处理所有帧的方法难以适用。为此,我们提出自回归变压器架构ART-STVG,将视频视为流式输入,逐帧处理,实现对长视频的高效处理。为建模时空上下文,设计空间与时间记忆库并应用于解码器;由于不同时刻的记忆并非总相关,引入简单有效的记忆选择策略,显著提升性能。此外,不同于并行的空间与时间定位,提出级联式时空设计,将空间解码器连接至时间解码器,利用细粒度空间线索辅助长视频中的复杂时间定位。在新扩展的LF-STVG数据集上实验表明,ART-STVG显著优于当前最先进方法,同时在常规短视频定位任务上也保持竞争力。代码已开源:https://github.com/HengLan/ART-STVG。

原文摘要 · Abstract (English)

In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG. Our code is at: https://github.com/HengLan/ART-STVG.

视频定位长视频自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。