用2D图像空间统一3D输入输出，让机器人操作学习更高效
BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models
- 将3D数据转为多视角2D图像，对齐视觉语言模型输入
- 用2D热图预测动作，在三个仿真基准上成功率提升至88.2%以上
- 仅需每任务3条轨迹即可高成功率泛化，适合样本稀缺场景
近期利用预训练视觉语言模型（VLM）构建视觉-语言-动作（VLA）模型，成为实现高效机器人操作学习的有前景方法。然而，现有方法极少将3D信号融入VLM进行动作预测，且未充分利用3D数据中的空间结构，导致样本效率低。本文提出BridgeVLA，一种新型3D VLA模型：(1) 将3D输入投影到多个2D图像，确保与VLM主干对齐；(2) 使用2D热图进行动作预测，统一输入与输出空间于一致的2D图像空间。此外，提出一种可扩展的预训练方法，使VLM主干在下游策略学习前具备预测2D热图的能力。大量实验表明，该方法能高效有效学习3D操作。在RLBench中，平均成功率从81.4%提升至88.2%；在COLOSSEUM中，复杂泛化设置下平均成功率从56.7%提升至64.0%；在GemBench中优于所有对比基线。真实机器人实验中，平均性能超越先进基线32%，在多种分布外场景（如视觉干扰、未见指令）下仍具鲁棒泛化能力。尤为突出的是，仅用每任务3条轨迹，即在10+任务上达到96.8%成功率，展现出极强样本效率。
原文摘要 · Abstract (English)
Recently, leveraging pre-trained vision-language models (VLMs) for building vision-language-action (VLA) models has emerged as a promising approach to effective robot manipulation learning. However, only few methods incorporate 3D signals into VLMs for action prediction, and they do not fully leverage the spatial structure inherent in 3D data, leading to low sample efficiency. In this paper, we introduce BridgeVLA, a novel 3D VLA model that (1) projects 3D inputs to multiple 2D images, ensuring input alignment with the VLM backbone, and (2) utilizes 2D heatmaps for action prediction, unifying the input and output spaces within a consistent 2D image space. In addition, we propose a scalable pre-training method that equips the VLM backbone with the capability to predict 2D heatmaps before downstream policy learning. Extensive experiments show the proposed method is able to learn 3D manipulation efficiently and effectively. BridgeVLA outperforms state-of-the-art baseline methods across three simulation benchmarks. In RLBench, it improves the average success rate from 81.4% to 88.2%. In COLOSSEUM, it demonstrates significantly better performance in challenging generalization settings, boosting the average success rate from 56.7% to 64.0%. In GemBench, it surpasses all the comparing baseline methods in terms of average success rate. In real-robot experiments, BridgeVLA outperforms a state-of-the-art baseline method by 32% on average. It generalizes robustly in multiple out-of-distribution settings, including visual disturbances and unseen instructions. Remarkably, it is able to achieve a success rate of 96.8% on 10+ tasks with only 3 trajectories per task, highlighting its extraordinary sample efficiency. Project Website:https://bridgevla.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。