arXiv:2602.13197cs.ROcs.CV2026-02被引 1

看人类视频学抓取,用仿真过滤出任务适配的抓法。

Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos

  • 用仿真给人类动作视频打标签,筛选出适合任务的抓取方式
  • 无需真实机器人数据,就能学会精准的抓取后操作技能
  • 适合无类人手的机器人,提升复杂任务的抓取成功率

通过观看人类视频学习操作技能,有望为机器人学习开辟大规模可扩展的数据来源。本文聚焦于需要先抓取再进行后续动作的预握操作(prehensile manipulation)。人类视频对学习抓取后的动作有很强信号,但对学习前置抓取行为帮助有限,尤其对不具备类人手的机器人而言。一个可行方案是采用模块化策略设计,使用专用抓取生成器产生稳定抓取。然而,任意稳定的抓取未必任务兼容,影响下游动作执行。为此,我们提出感知-仿真-模仿(Perceive-Simulate-Imitate, PSI)框架,利用仿真中配对的抓取-轨迹过滤处理人类视频动作数据,为轨迹数据添加抓取适用性标签,从而实现任务导向抓取能力的监督学习。实验证明,该框架可在不依赖任何真实机器人数据的情况下高效学习精确操作技能,相比直接使用抓取生成器,显著提升了鲁棒性表现。

原文摘要 · Abstract (English)

The ability to learn manipulation skills by watching videos of humans has the potential to unlock a new source of highly scalable data for robot learning. Here, we tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions. Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors, especially for robots without human-like hands. A promising way forward is to use a modular policy design, leveraging a dedicated grasp generator to produce stable grasps. However, arbitrary stable grasps are often not task-compatible, hindering the robot's ability to perform the desired downstream motion. To address this challenge, we present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data processed by paired grasp-trajectory filtering in simulation. This simulation step extends the trajectory data with grasp suitability labels, which allows for supervised learning of task-oriented grasping capabilities. We show through real-world experiments that our framework can be used to learn precise manipulation skills efficiently without any robot data, resulting in significantly more robust performance than using a grasp generator naively.

机器人学习视频模仿模块化策略仿真增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。