提升视频大模型时间推理能力,让机器看懂视频正反播放
Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs

- 分离视觉编码器与语言模型,追踪时间信息传递路径
- 发现投影层会破坏时间信号,时间保留型投影可显著提升性能
- 新架构超越人类在时间方向判断任务上的表现
时间之箭(AoT)任务要求判断视频是正放还是倒放,人类几乎能完美完成,但前沿视频大语言模型(Video-LLMs)仅略高于随机水平。我们通过隔离视觉编码器,追踪时间信息在编码器、投影器和语言模型间的流动,发现带有显式时序建模的视频中心编码器能有效捕捉时间信号,而帧中心编码器则不能。然而,当视频中心表示输入标准视频大模型架构后,性能常大幅下降,暴露出时间信息传递瓶颈。我们识别出投影器设计是关键因素:Q-Former会破坏时间信息,而保持时间不变的MLP投影能显著改善语言模型对时间信息的访问。分层分析显示编码器中时间表征的动态变化。基于此,我们构建了具备时间感知视频中心编码器、时间保留投影器和时间方向监督的新模型,在AoT$_{PPB}$任务上达到98.1%准确率,超越人类表现,并在更广泛的时序推理任务上提升最多6.0点(VITATECS-Direction)和1.3点(TVBench)。结果表明,视频大模型的时间推理需同时具备有效时序编码和可靠的信息传递。
原文摘要 · Abstract (English)
The Arrow-of-Time (AoT) task, determining whether a video plays forward or backward by recognizing temporal irreversibility, is one humans solve with near-perfect accuracy, yet frontier Video Large Language Models (Video-LLMs) perform only modestly above chance. This gap raises a key question: do visual backbones fail to encode temporal information, or does information bottleneck lie elsewhere in the Video-LLM architecture? We address this question by isolating the vision encoder from the Video-LLM and tracing temporal information across the encoder, projector, and LLM. We find that video-centric encoders with explicit temporal modeling encode strong temporal signals, whereas frame-centric encoders do not. However, when video-centric representations are passed through a standard Video-LLM architecture, performance often collapses, revealing a bottleneck of temporal information flow. We identify projector design as a key factor: Q-Former disrupts temporal information, while a time-preserved MLP projection substantially improves the LLM's access to such information. Our layer-wise analysis further shows temporal representation dynamics across encoder layers. Guided by these findings, we build a Video-LLM with temporal-aware video-centric encoder, time-preserved projector, and AoT supervision, surpassing human performance on AoT$_{PPB}$ with 98.1\% accuracy, and improving broader temporal reasoning tasks by up to 6.0 points on VITATECS-Direction and 1.3 points on TVBench. Our results show that temporal reasoning in Video-LLMs requires both effective temporal encoding and reliable transfer of this information to the LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。