用事件图结构增强视频推理的时空一致性,减少幻觉。
GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking
- 构建事件图捕捉视频内内外部事件关系,引导结构化推理。
- 在RexTime上提升4%以上定位准确率,降低9.8%时间序列幻觉。
- 适合需要高精度时序理解的视频问答与动作识别任务。
视频推理需要对视频中对象与事件间的细粒度时间依赖和事件级关系有深刻理解。当前多模态大模型在视频推理中容易出现严重的时间幻觉。其根本原因在于视觉-时间定位能力弱,且缺乏显式事件关系建模结构。模型常依赖密集描述等辅助文本,而非直接锚定在实际视觉证据上。这些文本表示本质上无结构,无法提供约束推理所需的因果关系。本文提出GraphThinker,一种强化微调方法,通过构建视频的结构化事件表示并强制视觉定位,联合降低推理幻觉。具体地,利用多模态大模型构建事件驱动的视频场景图(EVSG),捕捉事件内与事件间关系,引导结构化视频推理。同时,在强化学习微调中引入新型视觉注意力奖励,促使模型主动关注可靠视觉线索。在RexTime数据集上,GraphThinker在IoU=0.3下实现超过4%的定位准确率提升;在VidHalluc数据集上,相比现有最优方法,时间序列幻觉减少9.8%,动作幻觉相关二分类问答准确率提升7.6%。
原文摘要 · Abstract (English)
Video reasoning requires a fine-grained understanding of the temporal dependencies and event-level relations between objects and events in videos. Current Multimodal Large Language Models (MLLMs) are prone to severe temporal hallucinations in video reasoning. An underlying cause of these hallucinations is weak visual-temporal grounding and the lack of explicit structure for modelling event relations. Models often rely on auxiliary text, such as dense captions, rather than explicitly anchoring their reasoning in actual visual evidence. However, these textual representations are inherently unstructured and fail to provide explicit causal constraints needed to guide the model's reasoning. In this work, we propose GraphThinker, a reinforcement finetuning method that constructs a structured event representation of a video and enforces visual grounding to jointly reduce reasoning hallucinations. Specifically, we employ an MLLM to construct an Event-based Video Scene Graph (EVSG) that captures both intra- and inter-event relations, guiding a structured video reasoning process. Moreover, we address the weak grounding issue by introducing a novel visual attention reward during reinforcement finetuning that encourages the model to actively attend to reliable visual cues. On the RexTime dataset, GraphThinker achieves an over 4% improvement in IoU=0.3 for moment localisation. On the VidHalluc dataset, GraphThinker achieves a 9.8% improvement in reducing temporal sequence hallucination and a 7.6% gain in Binary QA in reducing action hallucination, compared to the state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。