为大模型智能体设计了评估记忆能力的新基准,填补了长期记忆评测空白。
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
- 基于认知科学构建多轮交互式评测框架,模拟增量信息积累过程。
- 覆盖准确回忆、实时学习、长程理解与选择性遗忘四大核心记忆能力。
- 适用于评估带外部记忆或工具集成的高级智能体,推动记忆机制研究。
近期大模型智能体评测主要关注推理、规划与执行能力,而关键组件——记忆(包括信息存储、更新与检索)因缺乏评测基准被忽视。我们称具备记忆机制的智能体为记忆智能体。本文基于记忆科学与认知科学经典理论,识别出记忆智能体的四大核心能力:准确回忆、测试时学习、长程理解与选择性遗忘。现有基准或受限于上下文长度,或仅适配静态长文本场景(如书籍问答),无法反映智能体在多轮交互中逐步累积信息的真实动态。且无一基准覆盖全部四类能力。为此,我们提出 MemoryAgentBench,将现有长上下文数据集重构为多轮格式,并引入新构建数据集,有效模拟记忆智能体的增量处理特性。通过精心选型与清洗,该基准全面覆盖上述四大核心能力,提供系统性、挑战性的评测环境。我们评估了从基础上下文依赖到检索增强生成(RAG)、外部记忆模块及工具集成的多种智能体。实证结果表明,当前方法尚无法掌握全部四项能力,凸显了发展综合性记忆机制的迫切需求。
原文摘要 · Abstract (English)
Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks. We term agents with memory mechanisms as memory agents. In this paper, based on classic theories from memory science and cognitive science, we identify four core competencies essential for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Existing benchmarks either rely on limited context lengths or are tailored for static, long-context settings like book-based QA, which do not reflect the interactive, multi-turn nature of memory agents that incrementally accumulate information. Moreover, no existing benchmarks cover all four competencies. We introduce MemoryAgentBench, a new benchmark specifically designed for memory agents. Our benchmark transforms existing long-context datasets and incorporates newly constructed datasets into a multi-turn format, effectively simulating the incremental information processing characteristic of memory agents. By carefully selecting and curating datasets, our benchmark provides comprehensive coverage of the four core memory competencies outlined above, thereby offering a systematic and challenging testbed for assessing memory quality. We evaluate a diverse set of memory agents, ranging from simple context-based and retrieval-augmented generation (RAG) systems to advanced agents with external memory modules and tool integration. Empirical results reveal that current methods fall short of mastering all four competencies, underscoring the need for further research into comprehensive memory mechanisms for LLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。