无需标注,从人类视频中学习精准抓取姿势。
RTAGrasp: Learning Task-Oriented Grasping from Human Videos via Retrieval, Transfer, and Alignment
- 从人类示范视频中提取抓取位置与方向约束,构建机器人记忆库。
- 通过语义匹配将人类抓取经验迁移至新物体,零训练完成抓取规划。
- 在真实机械臂上验证有效,适用于未见物体的抓取任务。
任务导向抓取(TOG)对机器人完成操作任务至关重要,需确定抓取位置与方向。现有方法要么依赖昂贵的人工标注,要么仅从人类示范中提取粗略抓取位置或区域,限制了其在真实场景中的实用性。为此,我们提出RTAGrasp,一个受人类抓取策略启发的检索、迁移与对齐框架。首先,该方法从人类抓取示范视频中轻松构建机器人记忆库,提取出抓取位置与方向约束;然后,在给定任务指令和目标物体视觉观测的前提下,从记忆库中检索最相似的人类抓取经验,并利用视觉基础模型的语义匹配能力,以无训练方式将抓取约束迁移到目标物体;最后,对齐转移后的抓取约束与机器人动作以执行。在公开的TOG基准数据集TaskGrasp上的评估显示,RTAGrasp在已见和未见物体类别上均表现出色,优于现有基线方法。真实世界实验进一步验证了其在机械臂上的有效性。代码、附录及演示视频可访问 https://sites.google.com/view/rtagrasp/home。
原文摘要 · Abstract (English)
Task-oriented grasping (TOG) is crucial for robots to accomplish manipulation tasks, requiring the determination of TOG positions and directions. Existing methods either rely on costly manual TOG annotations or only extract coarse grasping positions or regions from human demonstrations, limiting their practicality in real-world applications. To address these limitations, we introduce RTAGrasp, a Retrieval, Transfer, and Alignment framework inspired by human grasping strategies. Specifically, our approach first effortlessly constructs a robot memory from human grasping demonstration videos, extracting both TOG position and direction constraints. Then, given a task instruction and a visual observation of the target object, RTAGrasp retrieves the most similar human grasping experience from its memory and leverages semantic matching capabilities of vision foundation models to transfer the TOG constraints to the target object in a training-free manner. Finally, RTAGrasp aligns the transferred TOG constraints with the robot's action for execution. Evaluations on the public TOG benchmark, TaskGrasp dataset, show the competitive performance of RTAGrasp on both seen and unseen object categories compared to existing baseline methods. Real-world experiments further validate its effectiveness on a robotic arm. Our code, appendix, and video are available at \url{https://sites.google.com/view/rtagrasp/home}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。