用3D空间先验增强视觉语言动作模型,提升真实世界任务的泛化能力。
From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors
- 通过空间增强动作头注入丰富的3D空间标记,不修改主干网络
- 在三个仿真基准和十一个真实任务中达到最先进性能
- 仅需RGB图像即可获取强几何先验,支持多模态融合且无需重训练
现有视觉语言动作(VLA)模型虽在三维真实世界中执行任务,但通常基于二维编码器,导致空间推理能力不足,限制泛化与适应性。近期的3D整合方法或依赖专用传感器、跨模态迁移差,或注入弱线索,缺乏几何信息并损害视觉-语言对齐。本文提出FALCON(From Spatial to Action),将丰富3D空间标记注入动作头。FALCON利用空间基础模型从单张RGB图像提取强几何先验,并引入可选的具身空间模型,当有深度或姿态信息时可融合以提升精度,无需重训练或架构改动。空间标记由空间增强动作头处理,而非拼接至视觉-语言主干,从而保持语言推理能力。该设计有效解决了空间表示、模态迁移性和对齐问题。在三个仿真基准和十一个真实世界任务的综合评估中,FALCON表现卓越,持续超越对比基线,在杂乱环境、空间提示条件及物体尺度、高度变化下均保持鲁棒性。
原文摘要 · Abstract (English)
Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders, leaving a spatial reasoning gap that limits generalization and adaptability. Recent 3D integration techniques for VLAs either require specialized sensors and transfer poorly across modalities, or inject weak cues that lack geometry and degrade vision-language alignment. In this work, we introduce FALCON (From Spatial to Action), a novel paradigm that injects rich 3D spatial tokens into the action head. FALCON leverages spatial foundation models to deliver strong geometric priors from RGB alone, and includes an Embodied Spatial Model that can optionally fuse depth, or pose for higher fidelity when available, without retraining or architectural changes. To preserve language reasoning, spatial tokens are consumed by a Spatial-Enhanced Action Head rather than being concatenated into the vision-language backbone. These designs enable FALCON to address limitations in spatial representation, modality transferability, and alignment. In comprehensive evaluations across three simulation benchmarks and eleven real-world tasks, our proposed FALCON achieves state-of-the-art performance, consistently surpasses competitive baselines, and remains robust under clutter, spatial-prompt conditioning, and variations in object scale and height.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。