测试大模型在真实项目中的长期记忆能力，发现现有系统表现不佳。
RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
- 构建跨会话项目对话数据集，模拟真实任务演进过程。
- 超2000条对话显示当前记忆系统难以维持长期任务状态。
- 适合研究长程记忆、智能体协作与复杂任务管理的学者参考。
随着大型语言模型从静态对话界面演变为自主通用智能体，有效的记忆机制对保证长期一致性至关重要。然而，现有基准主要聚焦于闲聊或任务导向对话，未能捕捉‘长期项目导向’交互中智能体需追踪动态目标的需求。为此，我们提出首个基于真实项目场景的基准RealMem。RealMem包含11个场景下超过2,000条跨会话对话，采用自然用户查询进行评估。我们设计了一套合成流程，整合项目基础构建、多智能体对话生成及记忆与日程管理，以模拟记忆的动态演化。实验表明，当前记忆系统在处理真实项目中的长期状态和动态上下文依赖方面面临显著挑战。代码与数据集已开源。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture **"long-term project-oriented"** interactions where agents must track evolving goals. To bridge this gap, we introduce **RealMem**, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation. We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects. Our code and datasets are available at [https://github.com/AvatarMemory/RealMemBench](https://github.com/AvatarMemory/RealMemBench).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。