仅用摄像头图像训练机器人完成人机交接,无需真实试错。
Learning human-to-robot handovers through 3D scene reconstruction
- 用稀疏视角高斯点云重建场景,生成带动作的虚拟演示数据。
- 在16种日常物品上训练后直接部署到真实环境,成功率超85%。
- 适合想省去实机训练成本的研究者或工业应用团队。
从真实世界图像数据中学习机器人操作策略需要大量物理环境中的机器人动作试验。尽管仿真训练成本较低,但仿真与实际工作空间之间的视觉差异仍是主要瓶颈。最近的高斯点云可视化重建方法为机器人操作提供了新方向。本文提出首个仅依赖RGB图像、无需真实机器人训练或数据采集的监督式机器人交接方法。所提出的策略学习器——基于稀疏视角高斯点云重建的人机交接(H2RH-SGS),利用稀疏视角高斯点云对人机交接场景进行重建,生成由安装在机械臂上的相机捕捉的图像-动作配对的虚拟示范数据。由此,重建场景中模拟相机位姿的变化可直接转化为夹爪位姿的变化。我们在16种家用物品上收集示范数据并训练机器人策略,并直接部署于真实环境。在高斯点云重建场景和真实世界人机交接实验中均验证了该方法的有效性,证明其是人机交接任务的一种新型且高效的表示方式。
原文摘要 · Abstract (English)
Learning robot manipulation policies from raw, real-world image data requires a large number of robot-action trials in the physical environment. Although training using simulations offers a cost-effective alternative, the visual domain gap between simulation and robot workspace remains a major limitation. Gaussian Splatting visual reconstruction methods have recently provided new directions for robot manipulation by generating realistic environments. In this paper, we propose the first method for learning supervised-based robot handovers solely from RGB images without the need of real-robot training or real-robot data collection. The proposed policy learner, Human-to-Robot Handover using Sparse-View Gaussian Splatting (H2RH-SGS), leverages sparse-view Gaussian Splatting reconstruction of human-to-robot handover scenes to generate robot demonstrations containing image-action pairs captured with a camera mounted on the robot gripper. As a result, the simulated camera pose changes in the reconstructed scene can be directly translated into gripper pose changes. We train a robot policy on demonstrations collected with 16 household objects and {\em directly} deploy this policy in the real environment. Experiments in both Gaussian Splatting reconstructed scene and real-world human-to-robot handover experiments demonstrate that H2RH-SGS serves as a new and effective representation for the human-to-robot handover task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。