让机器人看懂任务指令，精准抓取物体的正确部位。
OVAL-Grasp: Open-Vocabulary Affordance Localization for Task Oriented Grasping
- 用大模型理解任务，视觉语言模型定位可操作区域
- 真实场景下抓对部位成功率78.3%，95%识别正确部位
- 支持遮挡和新物体，适合智能机器人作业
为在新型非结构化环境中操作物体，机器人需根据任务目标选择合适抓取部位。基于几何的方法常在视觉特征明显的部位、遮挡及未见物体上表现不佳。本文提出OVAL-Grasp，一种零样本开放词汇的任务导向抓取方法，结合大语言模型（LLM）与视觉语言模型（VLM），使机器人能依据给定任务在物体上准确识别应抓或避让的部位。输入RGB图像与任务描述后，系统由LLM判断目标部位，VLM进行分割，并生成物体上的2D可操作区域热力图。实验在20种家用物体上进行，每种物体执行3个不同任务，结果表明该方法优于两个基线方法。在实际测试中，对正确部位的识别率达95%，抓取准确率为78.3%；在部分遮挡条件下仍保持80%的部件选择成功率，且在依赖视觉特征的场景中表现良好。模块化设计经消融实验证明有效。项目主页：https://ekjt.github.io/OVAL-Grasp/
原文摘要 · Abstract (English)
To manipulate objects in novel, unstructured environments, robots need task-oriented grasps that target object parts based on the given task. Geometry-based methods often struggle with visually defined parts, occlusions, and unseen objects. We introduce OVAL-Grasp, a zero-shot open-vocabulary approach to task-oriented, affordance based grasping that uses large-language models and vision-language models to allow a robot to grasp objects at the correct part according to a given task. Given an RGB image and a task, OVAL-Grasp identifies parts to grasp or avoid with an LLM, segments them with a VLM, and generates a 2D heatmap of actionable regions on the object. During our evaluations, we found that our method outperformed two task oriented grasping baselines on experiments with 20 household objects with 3 unique tasks for each. OVAL-Grasp successfully identifies and segments the correct object part 95% of the time and grasps the correct actionable area 78.3% of the time in real-world experiments with the Fetch mobile manipulator. Additionally, OVAL-Grasp finds correct object parts under partial occlusions, demonstrating a part selection success rate of 80% in cluttered scenes. We also demonstrate OVAL-Grasp's efficacy in scenarios that rely on visual features for part selection, and show the benefit of a modular design through our ablation experiments. Our project webpage is available at https://ekjt.github.io/OVAL-Grasp/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。