让视频模型学会推断物体隐藏期间的状态变化,解决长期视频理解难题
StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videos

- 构建物体中心的时空状态记忆,追踪物体轨迹与关系演化
- 在长视频中将物体消失时的状态推理准确率提升至64.2(原39.6)
- 专为复杂状态推理设计,适合研究视频理解与推理的开发者
现有视觉语言模型在视频理解上表现优异,但在目标物体长时间不可见时,难以进行时空推理,常误将“不可见”视为“未知”。本文定义此问题为隐藏状态时空推理:通过上下文交互推断物体在长时间不可见期间的状态。为此提出StateTrace——一种新型物体中心框架,赋予视频大模型显式隐藏状态推理能力。该框架构建可复用的时空状态记忆,将物体轨迹、物间关系及状态转移事件结构化组织。推理时,检索与问题相关的状态演化路径,并转化为紧凑推理线索,使模型能明确回答物体为何消失、隐匿期间状态如何演变、当前状态是否持续。此外,构建了HSR-Bench诊断基准,包含1,427个来自1,384个独立视频的视频问答样本。跨多个VideoLLM的实验表明,StateTrace在公共基准和HSR-Bench上均显著提升性能(如VideoLLaMA3在HSR-Bench上从39.6提升至64.2)。
原文摘要 · Abstract (English)
Existing VLMs have achieved strong performance in video understanding, yet they struggle with long-video spatiotemporal reasoning when target objects become invisible, often mistaking "invisible" for "unknown". We define this challenge as hidden-state spatiotemporal reasoning: inferring object states during prolonged invisible intervals from context interactions. To address this, we propose StateTrace, a novel object-centric framework that endows VideoLLMs with an explicit mechanism for hidden state reasoning in long videos. StateTrace builds a reusable spatiotemporal state memory that organizes object trajectories, inter-object relations, and state-transition events into a structured reasoning substrate. At inference time, it retrieves question-relevant state-evolution trajectories and converts them into compact reasoning cues, enabling the model to explicitly reason about why an object disappears, how its state evolves while invisible, and whether that state should persist at query time. We further build HSR-Bench, a diagnostic benchmark for hidden-state reasoning, containing 1,427 video-QA samples from 1,384 unique videos. Extensive experiments across multiple VideoLLMs show that StateTrace consistently improves performance on both public benchmarks and HSR-Bench (e.g., improving VideoLLaMA3 from 39.6 to 64.2 on HSR-Bench).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。