测试大模型能否像人一样有结构地组织长期记忆。
Evaluating Memory Structure in LLM Agents
- 设计新基准 StructMemEval，评估模型组织记忆的能力。
- 简单检索增强模型在结构化任务中表现差，需提示才能完成。
- 当前大模型不主动识别记忆结构，需改进训练与框架。
基于大模型的智能体和聊天助手依赖长期记忆框架来存储可复用知识、回忆用户偏好并增强推理能力。随着记忆架构日益复杂，分析其能力并指导未来设计变得愈发困难。现有长期记忆评测多聚焦于简单事实记忆、多跳召回和时间相关变化，这些虽重要，但常可通过简单的检索增强大模型实现，无法检验复杂记忆层级结构。为此，我们提出 StructMemEval 基准，测试智能体组织长期记忆的能力，而非仅限于事实召回。我们收集了一组人类通过特定知识结构解决的任务：交易账本、待办清单、树状结构等。初步实验表明，简单检索增强的大模型在这些任务上表现不佳，而记忆智能体若获提示如何组织记忆则能可靠完成。然而，我们也发现现代大模型在未被提示时通常无法识别记忆结构。这揭示了未来大模型训练与记忆框架改进的重要方向。
原文摘要 · Abstract (English)
Modern LLM-based agents and chat assistants rely on long-term memory frameworks to store reusable knowledge, recall user preferences, and augment reasoning. As researchers create more complex memory architectures, it becomes increasingly difficult to analyze their capabilities and guide future memory designs. Most long-term memory benchmarks focus on simple fact retention, multi-hop recall, and time-based changes. While undoubtedly important, these capabilities can often be achieved with simple retrieval-augmented LLMs and do not test complex memory hierarchies. To bridge this gap, we propose StructMemEval - a benchmark that tests the agent's ability to organize its long-term memory, not just factual recall. We gather a suite of tasks that humans solve by organizing their knowledge in a specific structure: transaction ledgers, to-do lists, trees and others. Our initial experiments show that simple retrieval-augmented LLMs struggle with these tasks, whereas memory agents can reliably solve them if prompted how to organize their memory. However, we also find that modern LLMs do not always recognize the memory structure when not prompted to do so. This highlights an important direction for future improvements in both LLM training and memory frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。