看一次人类演示视频，机器人就能在不同环境下复现移动操作任务
HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation
- 通过AR眼镜采集人类视角视频，提取3D手部轨迹
- 将人手轨迹映射为机械臂末端的参考轨迹，支持跨环境复现
- 仅需一次示范即可在新场景中完成复杂操作，适合工业应用
我们提出一种新型人到机器人轨迹迁移系统，使机器人能通过学习人类演示视频完成物体操作。系统包含四个模块：第一是数据采集模块，利用AR头显从机器人视角收集人类演示视频；第二是视频理解模块，检测物体并从视频中提取3D人手轨迹；第三模块将人手轨迹转换为机器人末端执行器在三维空间中的参考轨迹；第四模块采用轨迹优化算法，求解机器人配置空间中可跟随该参考轨迹的运动路径。由此，机器人仅需观看一次人类演示视频，便可在不同环境中重复相同移动操作任务，即使物体位置与示范时不同。在移动操作机器人上对多种操作任务进行了实验，验证了系统的有效性。
原文摘要 · Abstract (English)
We introduce a novel system for human-to-robot trajectory transfer that enables robots to manipulate objects by learning from human demonstration videos. The system consists of four modules. The first module is a data collection module that is designed to collect human demonstration videos from the point of view of a robot using an AR headset. The second module is a video understanding module that detects objects and extracts 3D human-hand trajectories from demonstration videos. The third module transfers a human-hand trajectory into a reference trajectory of a robot end-effector in 3D space. The last module utilizes a trajectory optimization algorithm to solve a trajectory in the robot configuration space that can follow the end-effector trajectory transferred from the human demonstration. Consequently, these modules enable a robot to watch a human demonstration video once and then repeat the same mobile manipulation task in different environments, even when objects are placed differently from the demonstrations. Experiments of different manipulation tasks are conducted on a mobile manipulator to verify the effectiveness of our system
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。