机器人自动生成示范数据,直接从视觉图像学习抓取动作。
Self Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation

- 用自动生成的示范数据训练模型,无需人工标注或标定相机位置。
- 仿真中平面误差从9.69毫米降至5.38毫米,真实场景成功率超60%。
- 适合快速部署在缺乏专业设备的工业或实验室环境。
机器人抓取通常需要针对物体编程、手动标注数据或校准感知系统,限制了实际应用中的快速部署。基于示范的学习提供了更直接的替代方案,但收集示范仍需人工遥控或力控教学。本文提出一种自监督视觉抓取方法:机器人在目标位姿周围自动生成示范,并直接从腕部安装的RGB图像中学习相对位姿修正。所提流程使用ROS 2与Isaac Sim收集带标签的图像-位姿对,无需显式相机到机器人的外部标定。分别构建用于平面精调和粗略三维接近的数据集,训练卷积网络从单帧RGB图像回归相对平移与旋转。执行时,采用由粗到细的控制器:先使用含高度变化数据训练的模型接近物体,再用平面数据精调最终对齐。该方法在仿真和配备夹爪与单目相机的真实UR5e协作机器人上进行了评估。仿真中,精调阶段将最终平面偏差从9.69毫米降低至5.38毫米。真实实验中,系统对三个实物进行端到端抓取尝试,在无物体旋转条件下成功率达66.6%和63.6%,在旋转条件下仍保持部分鲁棒性。结果表明,自动生成的示范可支持低配置下的实用视觉抓取,同时揭示深度预测与物体依赖泛化仍存挑战。
原文摘要 · Abstract (English)
Robotic manipulation often requires object specific programming, manual data annotation, or calibrated perception pipelines, which limits rapid deployment in practical settings. Learning from demonstration offers a more direct alternative, but collecting demonstrations can still demand human teleoperation or kinesthetic teaching. This paper presents a self supervised visual manipulation method in which a robot automatically generates demonstrations around a target pose and learns relative pose corrections directly from wrist mounted RGB images. The proposed pipeline uses ROS~2 and Isaac Sim to collect labeled image-pose pairs without requiring explicit camera to robot extrinsic calibration. Separate datasets are generated for planar refinement and coarse three dimensional approach, and a convolutional network is trained to regress relative translation and rotation from single frame RGB observations. During execution, a coarse to fine controller first approaches the object using models trained with height variation and then refines the final alignment using planar data. The method is evaluated both in simulation and on a real UR5e collaborative robot equipped with a gripper and a monocular camera. In simulation, the refinement stage reduces the final planar dispersion from 9.69 mm to 5.38 mm. In real world experiments, the system performs end to end grasp attempts on three physical objects and reaches success rates of 66.6% and 63.6% for two objects without object rotation, while still maintaining partial robustness under rotated conditions. These results show that automatically generated demonstrations can support practical visual manipulation with limited setup effort, while also exposing remaining challenges in depth prediction and object dependent generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。