让机器人通过语言理解任意物体的可抓部位,实现快速精准抓取。
GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Grasping
- 用大模型直接在图像特征中预测可抓区域,跳过复杂3D建模
- 实测抓取成功率76.3%,推理速度比之前快29倍
- 适合需要快速适应新物体的人形机器人操作
基于人类描述推理任意物体的可抓部位,是机器人实现开放词汇操作的关键。现有抓取规划器受限于视觉-语言理解能力不足和耗时的3D辐射场建模,难以实现实时、开放词汇的交互。为此,我们提出GLOVER框架,通过微调大语言模型(LLMs)在RGB特征空间中预测物体可抓部位的视觉属性。我们构建了超过10,000张人-物交互图像数据集,标注统一的视觉与语言属性标签,支持多模态微调。GLOVER继承大模型的世界知识与常识推理能力,实现更细粒度的物体理解与复杂工具使用推理。为实现高效部署,我们提出无参数的抓取姿态估计方法AGE,将夹爪姿态对齐由可抓属性数据生成的超二次曲面。在30个真实桌面场景中评估,GLOVER在部位识别上达到86.0%成功率,在抓取上达76.3%;其属性推理速度比当前最优方案快约29倍,抓取姿态估计快40倍。我们还验证了跨机器人本体的泛化能力,证明其在具灵巧手人形机器人上的有效性。
原文摘要 · Abstract (English)
Inferring affordable (i.e., graspable) parts of arbitrary objects based on human specifications is essential for robots advancing toward open-vocabulary manipulation. Current grasp planners, however, are hindered by limited vision-language comprehension and time-consuming 3D radiance modeling, restricting real-time, open-vocabulary interactions with objects. To address these limitations, we propose GLOVER, a unified Generalizable Open-Vocabulary Affordance Reasoning framework, which fine-tunes the Large Language Models (LLMs) to predict the visual affordance of graspable object parts within RGB feature space. We compile a dataset of over 10,000 images from human-object interactions, annotated with unified visual and linguistic affordance labels, to enable multi-modal fine-tuning. GLOVER inherits world knowledge and common-sense reasoning from LLMs, facilitating more fine-grained object understanding and sophisticated tool-use reasoning. To enable effective real-world deployment, we present Affordance-Aware Grasping Estimation (AGE), a non-parametric grasp planner that aligns the gripper pose with a superquadric surface derived from affordance data. In evaluations across 30 table-top real-world scenes, GLOVER achieves success rates of 86.0% in part identification and 76.3% in grasping, with speeds approximately 29 times faster in affordance reasoning and 40 times faster in grasping pose estimation than the previous state-of-the-art. We also validate the generalization across embodiments, showing effectiveness in humanoid robots with dexterous hands.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。