评测大模型对视频中物体状态的持续追踪能力，发现其表现远低于人类。
Benchmarking Visual State Tracking in Multimodal Video Understanding
- 构建包含834段视频的VSTAT基准，需跨全程理解事件变化
- 顶尖模型在复杂任务上仅略高于随机猜测，远逊于人类表现
- 揭示模型擅长文字推理但视觉感知不足，适合研究多模态理解缺陷
理解视频不仅需要识别孤立片段，还需持续追踪实体、状态和事件。这种视觉状态追踪能力对视频理解至关重要，却在当前多模态大模型评估中被忽视。本文提出视觉状态追踪基准VSTAT，包含834个来自合成与真实视频的片段，搭配1500个需全程理解才能回答的问题，无法通过单帧或短片段解答。尽管顶尖多模态大模型在现有基准上表现优异，但在VSTAT上仍远低于人类，仅略高于答案先验基线。通过分析模型推理轨迹与视频流的匹配度，发现模型在文本推理上正确，但在视觉事件感知上失败。初步评估表明，近期基于代理的方法（如大模型视频代理、代码代理）也未能有效解决该问题，仍在VSTAT上表现不佳。
原文摘要 · Abstract (English)
Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based video agents and coding agents, do not readily resolve these failures, still falling short on VSTAT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。