用语义引导抓取,让机器人抓东西更符合实际用途。
SAGP: Semantic Affordance-Guided Grasp Planning via Coarse-Zone VLM Reasoning

- 将物体分区域后,用大模型判断各区域是否适合抓取。
- 在不依赖训练数据下,抓取成功率高且更符合功能需求。
- 特别适合抓壶、刀等有手柄或不对称的物品,无需精细分割。
基于几何的抓取规划虽能生成物理上有效的抓取,但忽略功能语义,常导致实际不合理的抓法,如握杯沿、握刀刃或瓶盖附近抓取,即便传统抓取指标达标,下游任务仍会失败。现有视觉语言模型(VLM)方法要么依赖细粒度类别特定部件分割,要么直接推断抓取姿态,后者易产生空间幻觉。因此,尚无实用的、无需训练的框架能可靠地将高层语义推理与几何抓取规划结合。本文提出语义可及性引导抓取规划(SAGP),一种基于粗粒度区域抽象的免训练流程。首先通过主成分分析对齐并利用距离驱动的DBSCAN聚类,将物体点云划分为顶部、中部、底部、侧边和凸起等空间区域,完全避开学习型分割。预训练的VLM通过结构化零样本查询评估各区域抓取质量,再融合几何、可达性和任务一致性信号,对反向对称抓取候选进行重排序。在PyBullet中使用Franka Panda机械臂对YCB物体的实验表明,SAGP在保持几何规划高成功率的同时,显著提升抓取的功能合理性,尤其在对称性差、带把手的物体上表现突出,而仅靠几何信息无法提供有效指导。提出的粗粒度区域抽象为基于VLM的推理与几何抓取规划之间提供了有效、免训练的桥梁,无需细粒度部件分割。
原文摘要 · Abstract (English)
Geometry-based grasp planners ensure physically valid grasps but ignore functional semantics, often generating grasps that are antipodal and collision-free yet practically inappropriate, for example, gripping a mug by its rim, a knife by the blade, or a bottle near its cap. These inconsistencies cause the downstream task to fail even when traditional grasp metrics are met. Existing vision-language model (VLM) approaches either depend on fine-grained, category-specific part segmentation or attempt to directly infer grasp poses, with the latter prone to spatial hallucinations. As a result, no practical, training-free framework has yet been proposed that robustly links high-level semantic reasoning to geometric grasp planning. We introduce Semantic Affordance-Guided Grasp Planning (SAGP), a training-free pipeline built on a coarse-zone abstraction layer. The method first partitions the object point cloud into spatial regions (top, middle, bottom, lateral sides, and protrusions) by applying PCA-based alignment followed by distance-driven DBSCAN clustering, entirely bypassing learned segmentation. A pre-trained VLM then assesses the grasp quality of each region through a structured zero-shot query, and the resulting zone-wise scores are fused with geometric, reachability, and task-alignment signals to re-rank antipodal grasp candidates. Experiments on YCB objects in PyBullet with a Franka Panda robot show that SAGP preserves the high success rate of geometry-only planning while substantially improving the functional appropriateness of selected grasps, particularly on asymmetric, handle-bearing objects where geometry alone is uninformative. The introduced coarse-zone abstraction offers an effective, training-free bridge between VLM-based reasoning and geometric grasp planning, without the need for fine-grained part segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。