让视频语言模型在内部用连续隐状态推理,不依赖外部工具或重生成视频。
STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models

- 用动态视频生成的隐状态引导模型内部推理,避免显式文字链式思考。
- 在多个基准上提升推理准确率,同时减少推理延迟和复杂度。
- 适合追求高效、低延迟视频理解的系统开发者使用。
许多视频推理任务需要追踪运动、时间顺序和跨帧的视觉状态演变。现有基于大视觉语言模型(LVLM)的方法常通过文本链式思维(CoT)、关键帧选择、重复插入帧或外部工具来实现,但这类方法会增加推理延迟和工程复杂性,并迫使时空证据被序列化为文本或反复从帧中重新编码。受‘视觉推理可在语言表达前隐式发生’的启发,我们提出STORMS(空间-时间推理通过内化建模),一个两阶段框架,使LVLM通过有界连续隐轨迹进行推理,而非显式的文本链式思维。第一阶段,通过生成视频得到的思维-视频表征对齐隐令牌,将隐状态扎根于动态视觉证据;第二阶段,仅使用答案监督进行训练,促使推理过程内化,无需步骤标注。生成思维视频仅用于训练;推理时,STORMS执行有界隐轨迹展开,无需重生成视频、插入帧或调用外部视觉工具。在VideoMME、MVBench、TempCompass和MMVU上的实验表明,与基于工具或视频生成的推理管道相比,STORMS在显著降低推理开销的同时提升了视频推理准确率。
原文摘要 · Abstract (English)
Many video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames. Existing methods built on large vision-language models (LVLMs) often address this challenge by externalizing reasoning through textual chain-of-thought (CoT), keyframe selection, repeated frame reinsertion, or external tool use. While effective, such pipelines increase inference-time latency and engineering complexity, and they force temporal-visual evidence to be serialized into text or repeatedly re-encoded from frames. Inspired by the intuition that visual reasoning can occur implicitly before verbalization, we propose STORMS (Spatial-Temporal reasOning via inteRnalized Modeling), a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT. In Stage I, STORMS aligns latent tokens with thought-video representations derived from generated videos, grounding the latent states in dynamic visual evidence. In Stage II, the model is further trained with answer-only supervision, encouraging the reasoning process to be internalized without step-by-step annotations. Generated thought videos are used only during training; at inference, STORMS performs a bounded latent rollout without regenerating videos, reinserting frames, or invoking external visual tools. Experiments on VideoMME, MVBench, TempCompass, and MMVU show that STORMS improves video reasoning accuracy while substantially reducing inference overhead compared with tool or video-generation-based reasoning pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。