让视觉语言模型通过4维动态想象理解空间变化。
4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

- 用连续隐空间模拟场景演化,实现内在动态想象。
- 在多个基准上超越基线,提升动态空间推理能力。
- 适合研究多模态推理与动态视觉理解的学者。
从单目视频进行动态空间推理对于连接视觉智能与物理世界至关重要,但对视觉语言模型(VLMs)仍具挑战。以往方法要么将时空推理完全转化为文本,导致冗长且不精确;要么依赖外部几何模块,增加推理复杂度却未增强模型内在能力。本文提出4DThinker,首个使VLM能够“以4D思考”的框架,即通过动态潜在心理图像,在连续隐藏空间中内化场景演化过程。我们首先构建可扩展、无需标注的数据生成流程,从原始视频合成4D推理数据;随后提出动态影像微调(DIFT),联合监督文本标记与4D隐变量,使模型锚定动态视觉语义。在此基础上,4D强化学习(4DRL)通过基于结果的奖励机制,仅将策略梯度限制在文本标记上,确保优化稳定。大量实验表明,4DThinker在多个动态空间推理基准上持续优于强基线,为VLM中的4D推理提供了新视角。代码已开源:https://github.com/zhangquanchen/4DThinker。
原文摘要 · Abstract (English)
Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。