用多智能体系统让机器人零样本理解语言指令抓取物体
GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System
- 三个专用智能体协作：规划、编码、观测，分步解析复杂指令
- 在两个大规模数据集上显著优于基线方法，实测表现优秀
- 无需微调即可适应新场景，适合真实世界人机交互应用
语言驱动的抓取检测有望革新人机交互，使机器人能根据自然语言指令执行抓取任务。然而现有方法面临两大挑战：一是难以解析复杂文本指令或在密集杂乱环境中表现不佳；二是多数方法需训练或微调才能适配新领域，限制了实际应用。本文提出GraspMAS，一种用于语言驱动抓取检测的多智能体系统框架。该框架通过三个专精智能体协同工作：Planner负责制定复杂查询策略，Coder生成并执行代码，Observer评估结果并提供反馈。在两个大规模数据集上的大量实验表明，GraspMAS显著优于现有基线方法。此外，仿真与真实机器人实验进一步验证了其有效性。
原文摘要 · Abstract (English)
Language-driven grasp detection has the potential to revolutionize human-robot interaction by allowing robots to understand and execute grasping tasks based on natural language commands. However, existing approaches face two key challenges. First, they often struggle to interpret complex text instructions or operate ineffectively in densely cluttered environments. Second, most methods require a training or finetuning step to adapt to new domains, limiting their generation in real-world applications. In this paper, we introduce GraspMAS, a new multi-agent system framework for language-driven grasp detection. GraspMAS is designed to reason through ambiguities and improve decision-making in real-world scenarios. Our framework consists of three specialized agents: Planner, responsible for strategizing complex queries; Coder, which generates and executes source code; and Observer, which evaluates the outcomes and provides feedback. Intensive experiments on two large-scale datasets demonstrate that our GraspMAS significantly outperforms existing baselines. Additionally, robot experiments conducted in both simulation and real-world settings further validate the effectiveness of our approach. Our project page is available at https://zquang2202.github.io/GraspMAS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。