arXiv:2607.12304cs.CVcs.LG2026-07

视频理解模型的评分可能骗人,它分不清是真懂时间顺序还是靠位置编码蒙混过关。

What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation

论文配图:What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
图 1 · 摘自论文原文
  • 提出反转下降法检测模型是否真看懂视频帧顺序
  • 发现主流模型多靠位置编码而非真实视觉顺序得分
  • 区分出两种模型:依赖位置和依赖视觉顺序,后者更可靠

视频时序问答基准的得分本应衡量模型对时间的理解能力,但实际混淆了两个问题:一是任务是否真的需要多帧及顺序,二是模型能否从像素中恢复顺序,还是仅依赖位置编码(RoPE)。大量分数其实源于单帧信息和答案先验。现有验证方法如帧乱序敏感性仅针对任务问题。本文提出无需标签的检测方法——反转下降(reversal-drop):当视觉序列反转而RoPE保持正向时,准确率下降程度可反映模型对真实顺序的依赖。该方法适用于已有基准。通过配对反向标签或确定性变换任务,可区分模型是跟随反转内容还是仅因冲突被破坏。Molmo2依赖位置编码,而Qwen3-VL真正读取视觉顺序。两者分别表现为位置主导与视觉序列主导。这一区分在两个基准、多个任务、不同规模下均成立,激活修补实验表明这是内部真实属性。两类模型在相反输入下失效,故相似总分不代表可互换,聚合分数无法反映潜在失败模式。

原文摘要 · Abstract (English)

A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. 1. The task question: is the question even temporal, does it need several frames and their order? and 2. The channel question, when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, a single frame and answer priors often carry it. The field's validity checks, frame-shuffle sensitivity and the accuracy gained from the full video, speak only to the task question. We contribute a label-free screen for the channel question, the reversal-drop: the accuracy lost when the visual sequence is reversed while RoPE remains forward. It can be applied to compatible temporal benchmarks without new annotations. Paired reverse labels, or tasks whose labels transform deterministically under reversal, distinguish models that follow reversed content from those merely disrupted by the conflict. Molmo2 answers the forward event reading order off positions, while Qwen3-VL answers the reversed event it actually sees, reading visual order (comparatively). We call them position-dominant and visual-sequence-dominant. The split holds across two benchmarks and several temporal tasks at two scales, and activation patching shows it is a real internal property, not an artifact of the conflict. The distinction matters, the two channels fail on opposite inputs so two models with similar score are not interchangable, i.e. an aggregate score does not reflect potential failure modes.

视频理解时序建模位置编码评估方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。