将视频时间理解转化为空间布局,提升定位精度。
T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding
- 用重叠滑窗将视频分段,帧按时间排成二维网格图
- 在多个基准上优于现有方法,显著提升定位准确率
- 适合需要精确时间定位的视频理解任务
视频时间定位(VTG)旨在找出与自然语言查询对应的时间片段,要求对复杂时间动态有全面理解。现有视觉-语言模型通常通过位置编码、基于文本的时间戳或视觉帧编号来感知时间动态。但这些方法存在明显缺陷:为每帧分配文本时间戳会增加计算开销并导致视觉注意力稀疏;位置编码难以捕捉绝对时间信息;视觉帧编号常牺牲空间细节。为此,我们提出时间到空间网格化(T2SGrid),将视频时间理解重构为一种空间理解任务。核心思想是按片段而非单帧处理视频内容,采用重叠滑动窗口机制将视频划分为时间片段。在每个窗口内,帧按时间顺序以行优先方式排列成复合网格图像,有效将时间序列转换为结构化的二维布局。该网格化不仅编码时间信息,还增强各网格内的局部注意力。此外,T2SGrid支持使用复合文本时间戳建立全局时间意识。在标准VTG基准上的实验表明,T2SGrid实现更优性能。
原文摘要 · Abstract (English)
Video Temporal Grounding (VTG) aims to localize the video segment that corresponds to a natural language query, which requires a comprehensive understanding of complex temporal dynamics. Existing Vision-LMMs typically perceive temporal dynamics via positional encoding, text-based timestamps, or visual frame numbering. However, these approaches exhibit notable limitations: assigning each frame a text-based timestamp token introduces additional computational overhead and leads to sparsity in visual attention, positional encoding struggles to capture absolute temporal information, and visual frame numbering often compromises spatial detail. To address these issues, we propose Temporal to Spatial Gridification (T2SGrid), a novel framework that reformulates video temporal understanding as a spatial understanding task. The core idea of T2SGrid is to process video content in clips rather than individual frames. we employ a overlapping sliding windows mechanism to segment the video into temporal clips. Within each window, frames are arranged chronologically in a row-major order into a composite grid image, effectively transforming temporal sequences into structured 2D layouts. The gridification not only encodes temporal information but also enhances local attention within each grid. Furthermore, T2SGrid enables the use of composite text timestamps to establish global temporal awareness. Experiments on standard VTG benchmarks demonstrate that T2SGrid achieves superior performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。