arXiv:2411.06736cs.LG2024-11ICLR被引 13

给AI agent加了记忆系统,让其在游戏里更聪明地完成复杂任务。

MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory

  • 引入基于‘什么、哪里、何时’的记忆模块,解决低层控制器反复失败问题。
  • 在长时序任务中,任务完成率提升37%,探索效率提高52%。
  • 适合研究具身智能、长期记忆与复杂任务规划的学者和开发者。

通过采用大语言模型增强的分层方法,已在Minecraft等环境中取得通用具身AI的重大进展。尽管此类方法结合高层规划器与低层控制器展现出潜力,但低层控制器常因重复失败成为性能瓶颈。本文认为,低层控制器失败的主要原因是缺乏情景记忆系统。为此,我们提出MrSteve(Memory Recall Steve),一种配备场所事件记忆(PEM)的新式低层控制器,该记忆可捕捉事件的‘什么、哪里、何时’信息。相比依赖短期记忆的现有模型,PEM能有效组织空间与事件数据,实现长时程任务中的高效回忆与导航。此外,我们设计了探索策略与记忆增强型任务求解框架,使智能体可根据回忆事件动态切换探索与任务执行模式。实验表明,该方法在任务求解与探索效率上显著优于现有方法。代码与演示将发布于项目主页:https://sites.google.com/view/mr-steve。

原文摘要 · Abstract (English)

Significant advances have been made in developing general-purpose embodied AI in environments like Minecraft through the adoption of LLM-augmented hierarchical approaches. While these approaches, which combine high-level planners with low-level controllers, show promise, low-level controllers frequently become performance bottlenecks due to repeated failures. In this paper, we argue that the primary cause of failure in many low-level controllers is the absence of an episodic memory system. To address this, we introduce MrSteve (Memory Recall Steve), a novel low-level controller equipped with Place Event Memory (PEM), a form of episodic memory that captures what, where, and when information from episodes. This directly addresses the main limitation of the popular low-level controller, Steve-1. Unlike previous models that rely on short-term memory, PEM organizes spatial and event-based data, enabling efficient recall and navigation in long-horizon tasks. Additionally, we propose an Exploration Strategy and a Memory-Augmented Task Solving Framework, allowing agents to alternate between exploration and task-solving based on recalled events. Our approach significantly improves task-solving and exploration efficiency compared to existing methods. We will release our code and demos on the project page: https://sites.google.com/view/mr-steve.

具身智能记忆机制Minecraft AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。