让机器人用手抓东西时既稳又懂指令,靠语义与接触信息联合指导。
SECOND-Grasp: Semantic Contact-guided Dexterous Grasping

- 用视觉语言推理预测抓取位置,再通过多视角一致性优化接触区域
- 在未见物体上抓取成功率达97.7%,语义理解抓取提升26.2%
- 适合需要精准语义理解的复杂抓取任务,如智能仓储、服务机器人
实现可靠的机器人操作(如灵巧抓取)需兼顾物理稳定性和语义任务引导,但二者常被割裂处理。本文提出SECOND-Grasp(语义接触引导的灵巧抓取)框架,使机械手在保证物理可行性的同时,根据语义推理动态调整抓取策略。首先通过视觉语言推理生成粗略接触提议,推断基于物体属性的接触点;接着通过分割定位多视角下的接触区域。为确保跨视角一致性,引入语义-几何一致性精化(SGCR),在保持语义一致性的前提下剔除几何无效区域,生成可靠的3D接触图。随后利用逆运动学为每个接触图求解可行手姿,作为策略学习的监督信号。模型在DexGraspNet数据集上训练,对已见和未见类别均表现优异,抓取成功率分别达98.2%和97.7%,且语义感知抓取性能提升12.8%和26.2%。进一步实验表明该方法在Shadow Hand和Allegro Hand等不同机械手上也具良好泛化性。
原文摘要 · Abstract (English)
Achieving reliable robotic manipulation, such as dexterous grasping, requires a synergy between physically stable interactions and semantic task guidance, yet these objectives are often treated as separate, disjoint goals. In this paper, we investigate how to integrate dexterous grasping techniques, i.e., physically stable grasps for object lifting and language-guided grasp generation, to achieve both physical stability and semantic understanding. To this end, we propose SECOND-Grasp (SEmantic CONtact-guided Dexterous Grasping), a unified framework that enables robotic hands to dynamically adjust grasping strategies based on semantic reasoning while ensuring physical feasibility. We begin by obtaining coarse contact proposals through vision-language reasoning to infer where contacts should occur based on object properties, followed by segmentation to localize these regions across views. To further ensure consistency across multiple viewpoints, we introduce Semantic-Geometric Consistency Refinement (SGCR), which refines initial contact predictions by enforcing semantic consistency across views and removing geometrically invalid regions, yielding reliable 3D contact maps. Then, we derive a feasible hand pose for each contact map via inverse kinematics, generating a supervision signal for policy learning. Our approach, trained on DexGraspNet, consistently outperforms baselines in lifting success rate on both seen and unseen categories, achieving 98.2% and 97.7%, respectively, while also improving intent-aware grasping by 12.8% and 26.2%. We further show promising results on additional datasets and robotic hands, including Shadow Hand and Allegro Hand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。