arXiv:2503.07511cs.ROcs.CV2025-03被引 110

让视觉语言动作模型用3D点云,不重训练就能提升空间推理能力。

PointVLA: Injecting the 3D World into Vision-Language-Action Models

  • 用轻量模块注入3D点云特征,只改无效网络层避免破坏原有模型
  • 20次演示即完成4个任务,实拍与图片能分辨,适配不同高度物体
  • 适合需要空间理解的机器人场景,如动态环境抓取、多任务泛化

视觉-语言-动作(VLA)模型通过大规模2D视觉-语言预训练在机器人任务中表现优异,但依赖RGB图像限制了其空间推理能力。重新训练模型使用3D数据成本过高,而丢弃现有2D数据又浪费资源。为此,我们提出PointVLA,一种无需重训练即可增强预训练VLA对点云输入支持的框架。该方法冻结原始动作专家,通过轻量级模块块注入3D特征。通过跳接块分析,定位原模型中效果较弱的模块,仅在此处注入3D信息,最大限度减少对预训练表示的干扰。大量实验表明,PointVLA在模拟和真实世界机器人任务中均优于OpenVLA、Diffusion Policy和DexVLA等先进2D模仿学习方法。关键优势包括:(1)少样本多任务能力,仅需每任务20次示范即可完成四项任务;(2)真实与图像对象区分能力,利用3D世界知识提升安全性和可靠性;(3)高度自适应性,可应对训练中未见的不同桌面高度物体。此外,PointVLA在长时序任务(如从移动传送带上拾取并打包物体)中也表现优异,展现出在复杂动态环境中的强泛化能力。

原文摘要 · Abstract (English)

Vision-Language-Action (VLA) models excel at robotic tasks by leveraging large-scale 2D vision-language pretraining, but their reliance on RGB images limits spatial reasoning critical for real-world interaction. Retraining these models with 3D data is computationally prohibitive, while discarding existing 2D datasets wastes valuable resources. To bridge this gap, we propose PointVLA, a framework that enhances pre-trained VLAs with point cloud inputs without requiring retraining. Our method freezes the vanilla action expert and injects 3D features via a lightweight modular block. To identify the most effective way of integrating point cloud representations, we conduct a skip-block analysis to pinpoint less useful blocks in the vanilla action expert, ensuring that 3D features are injected only into these blocks--minimizing disruption to pre-trained representations. Extensive experiments demonstrate that PointVLA outperforms state-of-the-art 2D imitation learning methods, such as OpenVLA, Diffusion Policy and DexVLA, across both simulated and real-world robotic tasks. Specifically, we highlight several key advantages of PointVLA enabled by point cloud integration: (1) Few-shot multi-tasking, where PointVLA successfully performs four different tasks using only 20 demonstrations each; (2) Real-vs-photo discrimination, where PointVLA distinguishes real objects from their images, leveraging 3D world knowledge to improve safety and reliability; (3) Height adaptability, Unlike conventional 2D imitation learning methods, PointVLA enables robots to adapt to objects at varying table height that unseen in train data. Furthermore, PointVLA achieves strong performance in long-horizon tasks, such as picking and packing objects from a moving conveyor belt, showcasing its ability to generalize across complex, dynamic environments.

3D感知机器人控制点云融合少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。