将视频转为带语义的3D空间代码，让大模型精准推理物理世界场景
Thinking with Spatial Code for Physical-World Video Reasoning
- 用统一架构将图像与几何特征融合，生成带方向的3D边界框和语义标签
- 在VSI-Bench上超越商用视觉语言模型，达到新基准性能
- 适合需要几何感知、空间推理的智能系统研发人员
我们提出Thinking with Spatial Code框架，将RGB视频转化为显式的时序一致3D表示，用于物理世界视觉问答。实验发现，所提出的空间编码器能将视频解析为结构化空间代码，包含显式的3D定向边界框和语义标签，使大语言模型（LLMs）可直接基于明确的空间变量进行推理。具体而言，我们设计空间编码器，通过融合6D物体解析与跟踪主干网络，结合几何预测实现图像与几何特征统一编码；并进一步使用基于空间规则的强化学习对LLM进行微调，奖励具有视角感知和几何基础的推理行为。结果表明，该模型在VSI-Bench数据集上超越现有商用视觉语言模型，创下新性能纪录。代码已开源：https://github.com/Beckschen/spatialcode。
原文摘要 · Abstract (English)
We introduce Thinking with Spatial Code, a framework that transforms RGB video into explicit, temporally coherent 3D representations for physical-world visual question answering. We highlight the empirical finding that our proposed spatial encoder can parse videos into structured spatial code with explicit 3D oriented bounding boxes and semantic labels, enabling large language models (LLMs) to reason directly over explicit spatial variables. Specifically, we propose the spatial encoder that encodes image and geometric features by unifying 6D object parsing and tracking backbones with geometric prediction, and we further finetuning LLMs with reinforcement learning using a spatial rubric reward that encourages perspective-aware, geometrically grounded inference. As a result, our model outperforms proprietary vision-language models on VSI-Bench, setting a new state-of-the-art. Code is available at https://github.com/Beckschen/spatialcode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。