让AI看视频时更懂时间,精准定位事件发生时刻。
Temporal-Aware Reasoning Optimization for Video Temporal Grounding

- 用密集字幕构建带时间戳的推理路径,提升探索效率。
- 通过时间敏感奖励机制,让模型学会判断推理是否合理。
- 适合需要精确时间定位的视频理解任务,如智能剪辑、搜索。
多模态大语言模型在视频时间定位任务中表现优异,但现有方法常生成浅层推理,难以指导精准定位。根本原因在于(1)随机探索效率低;(2)奖励函数仅关注答案正确性,忽略推理质量。为此,我们提出TaRO(Temporal-Aware Reasoning Optimization)框架:首先引入构造性推理探索,利用预生成的密集字幕构建基于视觉线索和时间戳的推理路径,实现高效高质量的时间感知推理探索;其次设计时间敏感奖励机制,若推理所依赖的事件边界被破坏,则推理路径的置信度应下降,以此作为推理质量的评判依据;最后采用渐进式课程策略,先用该奖励筛选优质推理路径,再进入自主生成阶段。实验表明,TaRO在多个视频时间定位基准上达到领先性能。代码已开源。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) have achieved remarkable progress in video temporal grounding with reinforcement learning for generating reasoning paths. However, existing models often produce superficial reasoning, which offers limited guidance for precise temporal localization. This limitation stems from (1) inefficient random exploration and (2) reward functions that focus solely on the answer correctness while ignoring reasoning quality. To address these issues, we propose TaRO (Temporal-Aware Reasoning Optimization), a framework that explicitly enhances the model's ability of thinking with time. First, we introduce a Constructive Reasoning Exploration that leverages pre-generated dense captions to construct reasoning paths grounded in explicit visual cues and timestamps, enabling efficient exploration of high-quality time-aware reasoning. Second, to evaluate reasoning quality, we design a Temporal-Sensitivity Reward. High-quality reasoning should be anchored to specific events and timestamps. If the event boundary under thinking is disrupted, such reasoning should become invalid, leading to a drop in the logit of the reasoning path. We utilize this drop as a critique of reasoning quality. Finally, TaRO follows a progressive curriculum, which starts by utilizing this reward to select better constructed reasoning paths, and evolves to a free exploration phase where the model autonomously generates effective reasoning. Experiments demonstrate that TaRO achieves state-of-the-art performance on VTG benchmarks. Code is available at https://github.com/oceanflowlab/TaRO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。