arXiv:2409.18313cs.ROcs.AI2024-09被引 60

让机器人像人一样记忆和检索环境信息,实现精准导航与描述。

Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation

  • 构建分层语义森林记忆系统,支持多尺度空间与语义信息存储。
  • 在千米级环境中成功处理250+导航与解释类查询,准确率显著提升。
  • 适用于各类机器人平台,适合需自主记忆的智能体研究者使用。

机器人探索与学习几乎无限,但所有知识必须可搜索、可调用。尽管语言领域中检索增强生成(RAG)已成为大规模非参数化知识的核心技术,但现有方法难以直接应用于具身领域——该领域具有多模态特性,数据高度相关,且感知需抽象。为此,我们提出Embodied-RAG框架,为具身智能体的基础模型引入非参数化记忆系统,可自主构建用于导航与语言生成的分层知识。该系统在不同环境与查询类型下支持从具体物体到整体氛围的多种分辨率。其核心记忆结构为语义森林,以不同粒度存储语言描述,实现跨平台上下文敏感输出。实验表明,Embodied-RAG有效将RAG拓展至机器人领域,在千米级环境中成功应对超过250个解释与导航查询,展现出作为通用非参数化系统的潜力。

原文摘要 · Abstract (English)

There is no limit to how much a robot might explore and learn, but all of that knowledge needs to be searchable and actionable. Within language research, retrieval augmented generation (RAG) has become the workhorse of large-scale non-parametric knowledge; however, existing techniques do not directly transfer to the embodied domain, which is multimodal, where data is highly correlated, and perception requires abstraction. To address these challenges, we introduce Embodied-RAG, a framework that enhances the foundational model of an embodied agent with a non-parametric memory system capable of autonomously constructing hierarchical knowledge for both navigation and language generation. Embodied-RAG handles a full range of spatial and semantic resolutions across diverse environments and query types, whether for a specific object or a holistic description of ambiance. At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail. This hierarchical organization allows the system to efficiently generate context-sensitive outputs across different robotic platforms. We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries across kilometer-level environments, highlighting its promise as a general-purpose non-parametric system for embodied agents.

具身智能检索增强机器人记忆多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。