让视频模型学会判断时间方向,提升事件理解能力。
ArrowGEV: Grounding Events in Video via Learning the Arrow of Time
- 区分时间敏感与不敏感事件,分别设计奖励机制。
- 在多个数据集上提升事件定位精度和时间方向识别率。
- 适合需要理解时间逻辑的视频分析任务使用。
视频中的事件定位是视频分析的基础能力。尽管视觉语言模型(VLMs)被广泛用于此任务,现有方法主要在正向视频中训练模型关联事件与时间戳,忽视了事件内在的时间结构和方向性,限制了模型的鲁棒性和泛化能力。受物理学中时间箭头的启发,我们提出ArrowGEV,一种通过强化学习显式建模事件时间方向性的框架,以提升VLMs对事件定位及时间方向性的理解。具体地,我们将事件分为时间敏感型(如放下包)和时间非敏感型(如左手握毛巾)。前者反转后意义显著改变,后者则保持语义不变。对于时间敏感事件,ArrowGEV引入奖励机制,鼓励模型区分正向与反向视频;对于时间非敏感事件,则要求在两个方向上保持一致的定位结果。大量实验表明,ArrowGEV不仅提升了事件定位精度和时间方向识别能力,还增强了模型的通用视频理解与推理能力。
原文摘要 · Abstract (English)
Grounding events in videos serves as a fundamental capability in video analysis. While Vision Language Models (VLMs) are increasingly employed for this task, existing approaches predominantly train models to associate events with timestamps in the forward video only. This paradigm hinders VLMs from capturing the inherent temporal structure and directionality of events, thereby limiting robustness and generalization. To address this limitation, inspired by the arrow of time in physics, which characterizes the intrinsic directionality of temporal processes, we propose ArrowGEV, a reinforcement learning framework that explicitly models temporal directionality in events to improve both event grounding and temporal directionality understanding in VLMs. Specifically, we categorize events into time-sensitive (e.g., putting down a bag) and time-insensitive (e.g., holding a towel in the left hand). The former denote events whose reversal substantially alters their meaning, while the latter remain semantically unchanged under reversal. For time-sensitive events, ArrowGEV introduces a reward that encourages VLMs to discriminate between forward and backward videos, whereas for time-insensitive events, it enforces consistent grounding across both directions. Extensive experiments demonstrate that ArrowGEV not only improves grounding precision and temporal directionality recognition, but also enhances general video understanding and reasoning ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。