arXiv:2605.07568cs.CVcs.CL2026-05

提升视频大模型时间推理能力,让机器看懂视频正反播放

Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs

论文配图:Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs
图 1 · 摘自论文原文
  • 分离视觉编码器与语言模型,追踪时间信息传递路径
  • 发现投影层会破坏时间信号,时间保留型投影可显著提升性能
  • 新架构超越人类在时间方向判断任务上的表现

时间之箭(AoT)任务要求判断视频是正放还是倒放,人类几乎能完美完成,但前沿视频大语言模型(Video-LLMs)仅略高于随机水平。我们通过隔离视觉编码器,追踪时间信息在编码器、投影器和语言模型间的流动,发现带有显式时序建模的视频中心编码器能有效捕捉时间信号,而帧中心编码器则不能。然而,当视频中心表示输入标准视频大模型架构后,性能常大幅下降,暴露出时间信息传递瓶颈。我们识别出投影器设计是关键因素:Q-Former会破坏时间信息,而保持时间不变的MLP投影能显著改善语言模型对时间信息的访问。分层分析显示编码器中时间表征的动态变化。基于此,我们构建了具备时间感知视频中心编码器、时间保留投影器和时间方向监督的新模型,在AoT$_{PPB}$任务上达到98.1%准确率,超越人类表现,并在更广泛的时序推理任务上提升最多6.0点(VITATECS-Direction)和1.3点(TVBench)。结果表明,视频大模型的时间推理需同时具备有效时序编码和可靠的信息传递。

原文摘要 · Abstract (English)

The Arrow-of-Time (AoT) task, determining whether a video plays forward or backward by recognizing temporal irreversibility, is one humans solve with near-perfect accuracy, yet frontier Video Large Language Models (Video-LLMs) perform only modestly above chance. This gap raises a key question: do visual backbones fail to encode temporal information, or does information bottleneck lie elsewhere in the Video-LLM architecture? We address this question by isolating the vision encoder from the Video-LLM and tracing temporal information across the encoder, projector, and LLM. We find that video-centric encoders with explicit temporal modeling encode strong temporal signals, whereas frame-centric encoders do not. However, when video-centric representations are passed through a standard Video-LLM architecture, performance often collapses, revealing a bottleneck of temporal information flow. We identify projector design as a key factor: Q-Former disrupts temporal information, while a time-preserved MLP projection substantially improves the LLM's access to such information. Our layer-wise analysis further shows temporal representation dynamics across encoder layers. Guided by these findings, we build a Video-LLM with temporal-aware video-centric encoder, time-preserved projector, and AoT supervision, surpassing human performance on AoT$_{PPB}$ with 98.1\% accuracy, and improving broader temporal reasoning tasks by up to 6.0 points on VITATECS-Direction and 1.3 points on TVBench. Our results show that temporal reasoning in Video-LLMs requires both effective temporal encoding and reliable transfer of this information to the LLM.

视频理解时间推理大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。