让机器人理解物体物理属性,更准地抓取复杂指令中的物品。
GraspCoT: Integrating Physical Property Reasoning for 6-DoF Grasping under Flexible Language Instructions
- 通过物理属性分步推理,把语言指令拆解成抓取步骤。
- 在真实场景中实现92.3%的抓取成功率,优于现有方法。
- 适合需要灵活理解口语指令的机器人应用开发。
柔性指令引导下的6-DoF抓取是现实机器人系统的重要挑战。现有方法依赖大语言模型(LLM)理解语义,但对物体物理属性的关注不足。本文提出GraspCoT框架,引入面向物理属性的思维链(CoT)推理机制,并通过辅助问答任务实现三阶段推理:目标解析、物理属性分析与抓取动作选择。该框架采用统一的多模态LLM架构,将3D场景的多视角观测编码为3D感知视觉标记,与CoT生成的文本标记共同嵌入模型,输出抓取位姿。此外,我们构建了IntentGrasp数据集,填补了多样化、间接口语指令下多物体抓取检测的公开数据空白。大量实验表明,该方法在IntentGrasp上表现优异,且在真实机器人上验证了实用性。代码已开源。
原文摘要 · Abstract (English)
Flexible instruction-guided 6-DoF grasping is a significant yet challenging task for real-world robotic systems. Existing methods utilize the contextual understanding capabilities of the large language models (LLMs) to establish mappings between expressions and targets, allowing robots to comprehend users' intentions in the instructions. However, the LLM's knowledge about objects' physical properties remains underexplored despite its tight relevance to grasping. In this work, we propose GraspCoT, a 6-DoF grasp detection framework that integrates a Chain-of-Thought (CoT) reasoning mechanism oriented to physical properties, guided by auxiliary question-answering (QA) tasks. Particularly, we design a set of QA templates to enable hierarchical reasoning that includes three stages: target parsing, physical property analysis, and grasp action selection. Moreover, GraspCoT presents a unified multimodal LLM architecture, which encodes multi-view observations of 3D scenes into 3D-aware visual tokens, and then jointly embeds these visual tokens with CoT-derived textual tokens within LLMs to generate grasp pose predictions. Furthermore, we present IntentGrasp, a large-scale benchmark that fills the gap in public datasets for multi-object grasp detection under diverse and indirect verbal commands. Extensive experiments on IntentGrasp demonstrate the superiority of our method, with additional validation in real-world robotic applications confirming its practicality. The code is available at https://github.com/cxmomo/GraspCoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。