让视频模型真正理解时间顺序，通过对比帧序变化来提升时序推理能力。
The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMs
- 在视频末帧的视觉表征层设计对比学习，强制模型捕捉事件动态
- 使用肯德尔τ距离衡量重排帧序的差异，引导模型关注时间变化
- 无需修改架构，适用于多种训练任务，显著提升时序理解性能
仅按顺序接收帧并不等于理解时间。现代视频大模型虽接收有序视频流，但主要监督信号作用于生成的文本，而非应承载事件动态的视频令牌表征。这种错配使模型可绕过内部视频表征，仅依赖物体、场景和语言先验即可回答时序问题。为此，我们提出VT-Contrast，一种基于表征层面的时间反事实目标。其设计聚焦于：时间监督应在何处进行？应暴露何种时间差异？该方法对选定的深层末帧视频令牌施加监督，并对比保持顺序的视图与同视频重排后的反事实视图，以肯德尔τ距离进行评分。该方法无需架构改动，兼容多种视频大模型训练任务，在多个时序理解基准上均提升整体性能。代码已公开于https://github.com/ANDgate99/VT-Contrast。
原文摘要 · Abstract (English)
Seeing frames in order does not mean representing time. Modern VideoLMs receive ordered video streams, yet their main supervision acts on generated text rather than video-token representations where event dynamics should first emerge. This mismatch allows models to learn temporal answers from shortcuts such as objects, scenes, and language priors, without requiring internal video representations to capture event progression. To address this, we propose VT-Contrast, a representation-level temporal counterfactual objective for VideoLMs. Its design asks where temporal supervision should act and what temporal differences it should expose. VT-Contrast supervises selected late-layer last-frame video tokens, where temporal information is expected to be integrated before language generation, and contrasts order-preserving views with same-video reordered counterfactuals graded by Kendall tau distance. It requires no architectural changes, is compatible with diverse VideoLM training tasks, and improves overall performance across temporal understanding benchmarks. Our code is available at https://github.com/ANDgate99/VT-Contrast.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。