通过深度感知提升视觉语言动作模型的空间推理能力
DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoning
- 引入预训练深度预测模块增强空间感知
- 在真实环境任务中达到78.5%成功率，优于基线65.0%
- 适合需要精准空间理解的机器人操作场景
视觉语言动作（VLA）模型虽展现出出色的泛化能力和语言引导操作能力，但在需要精确空间推理的任务中表现下降，因继承自视觉语言模型（VLM）的空间推理能力有限。现有VLA依赖大量动作数据预训练以在3D空间中对齐VLM，效率低且仍不足以实现准确的空间理解。本文提出DepthVLA，一种简单有效的VLA架构，通过预训练的深度预测模块显式引入空间意识。DepthVLA采用混合变压器设计，统一视觉语言模型、深度变换器与动作专家，并共享全部注意力机制，形成端到端模型，显著增强空间推理能力。在真实世界与模拟环境中的大量评估表明，DepthVLA优于当前最优方法：真实任务进度达78.5%（基线65.0%），LIBERO模拟器中为94.9%（基线93.6%），Simpler模拟器中为74.8%（基线58.8%）。代码将公开。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have recently shown impressive generalization and language-guided manipulation capabilities. However, their performance degrades on tasks requiring precise spatial reasoning due to limited spatial reasoning inherited from Vision-Language Models (VLMs). Existing VLAs rely on extensive action-data pretraining to ground VLMs in 3D space, which reduces training efficiency and is still insufficient for accurate spatial understanding. In this work, we present DepthVLA, a simple yet effective VLA architecture that explicitly incorporates spatial awareness through a pretrained depth prediction module. DepthVLA adopts a mixture-of-transformers design that unifies a VLM, a depth transformer, and an action expert with fully shared attentions, forming an end-to-end model with enhanced spatial reasoning. Extensive evaluations in both real-world and simulated environments show that DepthVLA outperforms state-of-the-art approaches, achieving 78.5% vs. 65.0% progress in real-world tasks, 94.9% vs. 93.6% in the LIBERO simulator, and 74.8% vs. 58.8% in the Simpler simulator. Our code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。