arXiv:2606.04436cs.CVcs.RO2026-06

让视觉语言动作模型隐式进行3D空间推理,无需3D传感器或显式文本生成。

3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

论文配图:3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training
图 1 · 摘自论文原文
  • 通过潜空间协同训练,分离几何感知与空间推理能力。
  • 在多个任务上达到顶尖性能,包括真实世界操作任务。
  • 仅用轻量适配器部署,不依赖3D模型或外部设备。

我们提出一种3D思维引导的协同训练框架,使视觉-语言-动作(VLA)模型在动作预测中隐式进行3D空间推理。核心思想是将3D几何感知与3D空间推理视为可解耦的能力,可在不同特征层次注入。训练时三个紧密耦合模块在潜空间协同工作:(1) 潜在3D几何感知模块将中间视觉特征对齐至3D基础模型,获取低层几何线索,无需修改VLM主干架构;(2) 在线3D推理蒸馏模块通过共享推理锚定标记缓解提示引发的推理差距,在3D VLM协同训练中,该标记作为首个输出标记以稳健编码空间先验;在VLA训练中,它作为输入标记插入任务与动作指令之间,将显式教师推理提示中的高层空间思维传递给学生动作提示,无需链式思维文本生成;(3) 这些解耦的几何与推理特征由空间增强的动作整合模块联合注入动作查询标记,作为分层空间条件防止动作捷径。部署时仅保留轻量适配器,丢弃3D基础模型与教师分支,完全基于2D图像运行,无需3D传感器、外部模型或显式文本生成,同时避免预训练VLM灾难性遗忘,在LIBERO、LIBERO-PLUS、SimplerEnv及真实世界操作任务上实现最先进性能。

原文摘要 · Abstract (English)

We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial reasoning are distinct capabilities that can be disentangled and injected at different feature hierarchies. During training, three tightly coupled components work in concert primarily within the latent space: (1) To gain geometric priors, a latent 3D geometry perception module aligns intermediate visual features with a 3D foundation model, acquiring low-level geometric cues without architectural modifications to the VLM backbone. (2) Complementing this, an online 3D reasoning distillation module mitigates the prompt-induced reasoning gap via a shared reasoning anchor token. During 3D VLM co-training, this anchor is emitted as the first output token to robustly encode spatial priors. During VLA training, it serves as an input token inserted between the task and action instructions, transferring high-level spatial thinking from explicit teacher reasoning prompts to student action prompts without chain-of-thought text generation. (3) These disentangled geometric and reasoning features are then united by a spatially augmented action integration, which jointly injects them into the action-query tokens as hierarchical spatial conditions to prevent action shortcuts. At deployment, our method retains only its lightweight adapters to perform implicit 3D reasoning, discarding the 3D foundation model and the teacher branch used for supervision. Consequently, it operates purely on 2D images without 3D sensors, external models, or explicit text generation while preventing catastrophic forgetting of the pretrained VLM, achieving state-of-the-art performance on LIBERO, LIBERO-PLUS, SimplerEnv, and real-world manipulation tasks.

3D推理视觉语言动作协同训练空间先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。