构建可评估记忆能力的Minecraft智能体任务集,支持真实交互与可验证结果。
MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents
- 通过专家共玩收集真实任务,生成带条件和依赖结构的参数化模板。
- 在8名玩家上测试216个子任务,发现代码执行与物品管理是主要失败点。
- 适合研究具身智能体记忆机制、混合意图交互的开发者与研究者使用。
我们提出MineNPC-Task,一个由用户创建的基准测试框架,用于评估开放世界Minecraft中具备记忆能力的混合意图大模型智能体。任务通过与专家玩家的形成性与总结性共玩获取,经标准化为带有明确前置条件和依赖关系的参数化模板。任务配备机器可验证的校验器,在限定知识范围内禁止外部捷径。该框架记录计划、动作与记忆事件,包括计划预览、针对性澄清、记忆读写、前置条件检查及修复尝试,并仅基于游戏内证据报告相对于总尝试子任务的完成情况。作为初步成果,我们使用GPT-4o在8名经验玩家中评估了216个子任务。观察到代码执行、物品库存与工具处理、引用和导航中的重复失败模式,同时发现混合意图澄清和轻量记忆使用可支持成功恢复。参与者对交互质量和界面可用性评价积极,但指出跨任务记忆持久性不足。我们公开完整任务集、校验器、日志与评估框架,以支持未来记忆感知具身智能体的透明与可复现评估。
原文摘要 · Abstract (English)
We present MineNPC-Task, a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world Minecraft. Rather than relying on synthetic prompts, tasks are elicited through formative and summative co-play with expert players, then normalized into parametric templates with explicit preconditions and dependency structure. These tasks are paired with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. The harness captures plan, action, and memory events, including plan previews, targeted clarifications, memory reads and writes, precondition checks, and repair attempts, and reports outcomes relative to the total number of attempted subtasks using only in-world evidence. As an initial snapshot, we instantiate the framework with GPT-4o and evaluate 216 subtasks across 8 experienced players. We observe recurring breakdown patterns in code execution, inventory and tool handling, referencing, and navigation, alongside successful recoveries supported by mixed-initiative clarifications and lightweight memory use. Participants rated interaction quality and interface usability positively, while noting the need for stronger memory persistence across tasks. We release the complete task suite, validators, logs, and evaluation harness to support transparent and reproducible evaluation of future memory-aware embodied agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。