用视觉轨迹统一表示人与机器人的动作,仅需少量视频就能教会机器人完成日常任务。
Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning
- 将动作表示为图像上的短时2D运动轨迹,实现人机动作空间统一
- 仅需数分钟人类视频和少量机器人示范,实测成功率高达86.5%
- 支持多视角合成6自由度轨迹,可泛化到仅在人类视频中出现的场景
让机器人自主完成日常任务仍是挑战。模仿学习(IL)通过示范赋予机器人技能,但依赖人工操控机器人数据,成本高昂。人类视频提供可扩展替代方案,却因缺乏机器人动作标签难以直接用于训练。为此,我们提出将动作表示为图像上的短时2D轨迹(运动轨迹),捕捉手或机械臂末端的运动方向。我们构建了名为运动轨迹策略(MT-pi)的模仿学习策略,接收图像输入并输出运动轨迹作为动作。通过这一统一的跨体感动作空间,MT-pi仅需几分钟人类视频和有限机器人示范即可完成任务。测试时,从双摄像头视角预测运动轨迹,并通过多视角融合恢复6自由度轨迹。在4个真实任务中,平均成功率达86.5%,比不使用人类数据或该动作空间的前沿基线高出40%,且能泛化至仅在人类视频中出现的场景。代码与视频见官网 https://portal-cornell.github.io/motion_track_policy/。
原文摘要 · Abstract (English)
Teaching robots to autonomously complete everyday tasks remains a challenge. Imitation Learning (IL) is a powerful approach that imbues robots with skills via demonstrations, but is limited by the labor-intensive process of collecting teleoperated robot data. Human videos offer a scalable alternative, but it remains difficult to directly train IL policies from them due to the lack of robot action labels. To address this, we propose to represent actions as short-horizon 2D trajectories on an image. These actions, or motion tracks, capture the predicted direction of motion for either human hands or robot end-effectors. We instantiate an IL policy called Motion Track Policy (MT-pi) which receives image observations and outputs motion tracks as actions. By leveraging this unified, cross-embodiment action space, MT-pi completes tasks with high success given just minutes of human video and limited additional robot demonstrations. At test time, we predict motion tracks from two camera views, recovering 6DoF trajectories via multi-view synthesis. MT-pi achieves an average success rate of 86.5% across 4 real-world tasks, outperforming state-of-the-art IL baselines which do not leverage human data or our action space by 40%, and generalizes to scenarios seen only in human videos. Code and videos are available on our website https://portal-cornell.github.io/motion_track_policy/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。