让模型在训练中学会‘预想’未来画面,推理时直接出答案,更快更准。
Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

- 训练时让模型预测未来帧的隐向量,内化视觉推理能力。
- 相比显式生成图像,推理速度提升5倍以上,准确率不降反升。
- 适合需要实时响应的视频理解任务,如自动驾驶、机器人决策。
多模态大模型常通过生成中间推理图像来实现视觉链式思考(Visual CoT),以理解空间、时间与具身环境。然而该方法带来显著推理开销,尤其影响主动式视频推理。本文提出内部化视觉思维(IVT),一种后训练框架,在无标签视频上联合优化文本预测与未来嵌入预测。给定部分观测视频,IVT同时预测未来帧的潜在表示与目标文本答案,促使模型捕捉运动、物体变化、交互及潜在意图。推理时无需生成或重编码未来帧,直接输出答案。在六种设置下控制实验验证,IVT优于直接微调,且保持相同推理路径。相比显式视觉链式思考,性能相当或更优,平均端到端延迟降低超过5倍。结果表明,主动视频推理无需在推理时显式生成像素级图像,预测性世界建模可在训练中内化,从而构建更准确、更高效的多模态推理系统。
原文摘要 · Abstract (English)
Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-embedding prediction over unlabeled videos. Given a partially observed video, IVT predicts latent representations of future frames together with the target textual answer, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames. We conduct controlled studies across target representations, decoder designs, prediction horizons, data mixtures, training curricula, and predictive objectives. IVT improves over direct-answer fine-tuning on all six evaluation settings while retaining the same inference pathway. Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x. Together, our findings suggest that explicit pixel-space generation at inference time, as used in visual chain-of-thought, may not be necessary for effective proactive video reasoning. Predictive world modeling can be internalized during training to produce multimodal reasoners that are both more accurate and substantially more efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。