用关键点建模物体功能接触,让机械手精准抓取复杂工具。
Multi-Keypoint Affordance Representation for Functional Dexterous Grasping
- 通过多关键点表示直接编码任务驱动的抓取姿态。
- 在真实数据集和模拟环境中,抓取准确率与一致性显著提升。
- 无需人工标注,适合机器人抓取、人机协作等场景。
功能性灵巧抓取需精确的人-物交互,现有基于能力的方法主要预测粗粒度交互区域,无法直接约束抓取姿态,导致视觉感知与操作脱节。为此,我们提出一种多关键点能力表示方法,通过定位功能接触点,直接编码任务驱动的抓取配置。方法引入接触引导的多关键点能力(CMKA),利用人类抓握经验图像进行弱监督,并结合大视觉模型提取精细能力特征,实现泛化且避免人工关键点标注。此外,提出基于关键点的抓取矩阵变换(KGT)方法,确保手部关键点与物体接触点的空间一致性,从而建立视觉感知与灵巧操作之间的直接联系。在公开的真实世界FAH数据集、IsaacGym仿真环境及挑战性机器人任务上的实验表明,该方法显著提升了能力定位精度、抓取一致性,并实现了对未见过的工具和任务的泛化能力,弥合了视觉能力学习与灵巧机器人操作之间的鸿沟。源代码与演示视频已开源:https://github.com/PopeyePxx/MKA。
原文摘要 · Abstract (English)
Functional dexterous grasping requires precise hand-object interaction, going beyond simple gripping. Existing affordance-based methods primarily predict coarse interaction regions and cannot directly constrain the grasping posture, leading to a disconnection between visual perception and manipulation. To address this issue, we propose a multi-keypoint affordance representation for functional dexterous grasping, which directly encodes task-driven grasp configurations by localizing functional contact points. Our method introduces Contact-guided Multi-Keypoint Affordance (CMKA), leveraging human grasping experience images for weak supervision combined with Large Vision Models for fine affordance feature extraction, achieving generalization while avoiding manual keypoint annotations. Additionally, we present a Keypoint-based Grasp matrix Transformation (KGT) method, ensuring spatial consistency between hand keypoints and object contact points, thus providing a direct link between visual perception and dexterous grasping actions. Experiments on public real-world FAH datasets, IsaacGym simulation, and challenging robotic tasks demonstrate that our method significantly improves affordance localization accuracy, grasp consistency, and generalization to unseen tools and tasks, bridging the gap between visual affordance learning and dexterous robotic manipulation. The source code and demo videos are publicly available at https://github.com/PopeyePxx/MKA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。