让机器人看懂3D场景，学会没练过的任务
3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks
- 引入深度感知与区域聚焦，增强3D空间理解能力
- 在未见任务上成功率提升8.8%，达98.1%
- 适合想提升机器人泛化能力的研究者
3D机器人操作需精确计算多自由度关节轨迹以实现稳定控制。为此，机器人必须融合语义理解与视觉感知，将真实世界观测转化为低层控制指令。近年来，视觉-语言-动作（VLA）模型通过将RGB图像与语言指令映射到任务空间速度，展现出巨大潜力，通常基于大规模遥控示范数据训练。然而，这些模型在分布外任务上泛化能力有限。本文提出3D-CAVLA，一种新型微调框架，通过三个关键组件提升VLA策略的泛化能力：(i) 链式思维推理实现结构化决策，(ii) 深度感知感知实现3D空间理解，(iii) 任务导向的感兴趣区域检测实现精准操作。在LIBERO仿真环境中的大量实验表明，3D-CAVLA在多种域内任务中平均成功率达98.1%。在未见任务上，其成功率绝对提升8.8%，证明了3D场景感知对鲁棒泛化的价值。我们在真实桌面上验证了该方法，结果表明模型可有效从仿真迁移到物理机器人。3D-CAVLA实现训练收敛速度提升3倍以上，真实世界未见任务成功率提高25%。代码与未见任务数据集将开源：https://3d-cavla.github.io
原文摘要 · Abstract (English)
Robotic manipulation in 3D requires effective computation of N degree-of-freedom joint-space trajectories that enable precise and robust control. To achieve this, robots must integrate semantic understanding with visual perception to transform real-world observations into low-level control for object interaction. Recent advances in Vision-Language-Action (VLA) models have shown promise by mapping RGB images and language instructions to task space velocities, typically trained on large datasets of teleoperated demonstrations. However, these models often struggle with generalization beyond their training distributions. In this work, we introduce 3D-CAVLA, a novel finetuning framework that enhances task generalization of VLA policies by incorporating three key components: (i) chain-of-thought reasoning for structured decision-making, (ii) depth-aware perception for 3D spatial understanding, and (iii) task-oriented region-of-interest detection for focused manipulation. Extensive experiments in the LIBERO simulation environment demonstrate that 3D-CAVLA achieves an average success rate of 98.1% across diverse in-domain task suites. On unseen tasks, 3D-CAVLA delivers an absolute improvement of 8.8% in success rate, underscoring the benefits of 3D scene awareness for robust generalization. We validate our approach on real-world tabletop experiments demonstrating that the proposed model translates effectively from simulation to physical robots. 3D-CAVLA achieves over a 3X faster training convergence and delivers a 25% gain in success rate on unseen real world tasks. We will open-source our code and the unseen tasks dataset to promote community-driven research here: https://3d-cavla.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。