arXiv:2607.13305cs.CVcs.AI2026-07中稿 · , ACM Internationa…

视频大模型的准确率未必代表真懂视觉,研究发现它可能只是在瞎猜。

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

论文配图:Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
图 1 · 摘自论文原文
  • 用黑屏对比原视频,测出模型是否真依赖视觉信息。
  • 多数模型在黑屏下仍能答对,说明它们没真正看懂视频。
  • 适合关注视频理解真实能力的研究者和评测人员。

视频大语言模型的基准测试准确率常被当作视觉理解的证据。本研究审计了20个参数量从2亿到780亿、涵盖10种架构的模型。提出视觉依赖差距(VDG),即原视频与黑屏条件下问答正确率之差。在MVBench上配对麦克内马尔检验显示:模型在原视频下表现差异显著(p=0.0003),但在黑屏下无差异(p=0.53),说明准确率与视觉依赖可分离。任务类型排名稳定:属性感知强依赖视觉,而时序推理接近纯语言基线。从黑屏到单帧、乱序帧再到原视频的诊断阶梯表明,帧多样性贡献主要视觉收益,时间顺序几乎不提升准确率(16个开源模型)。从0.5到24帧/秒的消融实验排除了稀疏采样的影响。H.264实验揭示稳定准确率下存在双向问题级答案翻转。该诊断方法也适用于4个API模型,其VDG值在0.025至0.315之间。结果支持将VDG作为衡量视频模型是否具备视觉根基的标准。代码已公开。

原文摘要 · Abstract (English)

Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.

视频理解大模型评测视觉依赖基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。