仅用一段人类操作视频,让机器人学会通用工具操作。
MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondence
- 构建功能坐标系,通过关键点抽象实现跨工具功能对应。
- 仅需一次观看视频,即可在新工具上完成相似功能操作。
- 适合需要快速迁移技能的机器人学习场景。
从人类视频中模仿工具操作为机器人教学提供了直观途径,同时可作为人力密集型遥操作数据采集的高效替代方案。尽管人类仅需观察一次即可轻松将技能迁移到功能相似的不同工具上,当前机器人仍难以实现此类泛化。核心挑战在于建立功能层面的对应关系,需应对功能相近工具间的显著几何差异(即同类内差异)。为此,我们提出MimicFunc框架,利用基于关键点抽象构建的功能帧(function frame)建立功能对应,实现工具操作技能的模仿。实验表明,MimicFunc能有效使机器人从单一RGB-D人类视频中泛化技能,成功操控新型工具完成功能等价任务。此外,借助其一次性泛化能力,生成的轨迹可直接用于训练视觉-运动策略,无需为新物体收集繁琐的遥操作数据。代码与视频见https://sites.google.com/view/mimicfunc。
原文摘要 · Abstract (English)
Imitating tool manipulation from human videos offers an intuitive approach to teaching robots, while also providing a promising and scalable alternative to labor-intensive teleoperation data collection for visuomotor policy learning. While humans can mimic tool manipulation behavior by observing others perform a task just once and effortlessly transfer the skill to diverse tools for functionally equivalent tasks, current robots struggle to achieve this level of generalization. A key challenge lies in establishing function-level correspondences, considering the significant geometric variations among functionally similar tools, referred to as intra-function variations. To address this challenge, we propose MimicFunc, a framework that establishes functional correspondences with function frame, a function-centric local coordinate frame constructed with keypoint-based abstraction, for imitating tool manipulation skills. Experiments demonstrate that MimicFunc effectively enables the robot to generalize the skill from a single RGB-D human video to manipulating novel tools for functionally equivalent tasks. Furthermore, leveraging MimicFunc's one-shot generalization capability, the generated rollouts can be used to train visuomotor policies without requiring labor-intensive teleoperation data collection for novel objects. Our code and video are available at https://sites.google.com/view/mimicfunc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。