arXiv:2410.23968cs.RO2024-10被引 30

用动态子图检索提升机器人规划效率,解决大场景下语言模型输入瓶颈。

EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planning

  • 基于检索增强生成思想,动态提取任务相关3D场景子图。
  • 输入令牌数减少一个数量级,每步规划时间降低70%。
  • 适用于移动机械臂与四足机器人的实际部署,提升成功率。

大型语言模型(LLMs)的进步推动了真实开放环境中的机器人规划发展。3D场景图(3DSGs)因其紧凑且语义丰富,成为支撑此类规划器的理想环境表示。然而,随着环境规模扩大(如追踪实体数量增加)和场景图信息复杂度上升(如维护更多属性),直接将完整3DSG输入到基于LLM的规划器会因输入令牌数限制和注意力偏差而变得不可行。受检索增强生成(RAG)方法的启发,我们提出一种面向具身领域的3D场景子图检索框架——EmbodiedRAG,用于增强基于LLM的机器人规划能力。该框架可随环境变化及任务相关性动态调整检索内容。实验表明,EmbodiedRAG能显著减少输入令牌数(降低一个数量级),并将每步规划时间平均缩短70%,同时在AI2Thor模拟家庭环境中使用单臂移动机械手完成任务的成功率得到提升。此外,我们在配备机械臂的四足机器人上实现EmbodiedRAG,验证其在真实环境边缘部署中的性能优势。

原文摘要 · Abstract (English)

Recent advances in Large Language Models (LLMs) have helped facilitate exciting progress for robotic planning in real, open-world environments. 3D scene graphs (3DSGs) offer a promising environment representation for grounding such LLM-based planners as they are compact and semantically rich. However, as the robot's environment scales (e.g., number of entities tracked) and the complexity of scene graph information increases (e.g., maintaining more attributes), providing the 3DSG as-is to an LLM-based planner quickly becomes infeasible due to input token count limits and attentional biases present in LLMs. Inspired by the successes of Retrieval-Augmented Generation (RAG) methods that retrieve query-relevant document chunks for LLM question and answering, we adapt the paradigm for our embodied domain. Specifically, we propose a 3D scene subgraph retrieval framework, called EmbodiedRAG, that we augment an LLM-based planner with for executing natural language robotic tasks. Notably, our retrieved subgraphs adapt to changes in the environment as well as changes in task-relevancy as the robot executes its plan. We demonstrate EmbodiedRAG's ability to significantly reduce input token counts (by an order of magnitude) and planning time (up to 70% reduction in average time per planning step) while improving success rates on AI2Thor simulated household tasks with a single-arm, mobile manipulator. Additionally, we implement EmbodiedRAG on a quadruped with a manipulator to highlight the performance benefits for robot deployment at the edge in real environments.

机器人规划3D场景图RAG具身智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。