评测大模型在多实体动态记忆中的推理能力，发现现有系统普遍失效。
MEME: Multi-entity & Evolving Memory Evaluation
- 构建六项任务，覆盖多实体与动态变化的完整评估空间。
- 依赖推理任务平均准确率仅3%（Cascade）和1%（Absence），严重退化。
- 高成本配置才能部分缓解问题，当前方案难规模化应用。
基于大语言模型的智能体日益在持久环境中运行，需跨会话存储、更新并推理信息。现有基准仅评估单实体更新，而MEME定义了六个任务，涵盖多实体与演化维度的完整空间，包括三项此前未被评分的任务：级联（Cascade）与缺失（Absence）（依赖推理）以及删除（Deletion）（移除后状态）。在100个受控实验中评估六种内存系统，覆盖三种内存范式，发现所有系统在默认配置下依赖推理任务表现崩溃（Cascade平均准确率3%，Absence为1%），尽管静态检索性能良好。提示优化、深度检索、减少干扰噪声及更强的LLM均无法弥合差距。仅有文件型智能体搭配Claude Opus 4.7作为内部模型部分缓解问题，但成本达基线的70倍，表明当前解决方案难以规模化。代码与数据已发布于项目主页：https://seokwonjung-jay.github.io/meme-eval/。
原文摘要 · Abstract (English)
LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 as its internal LLM partially closes the gap, but at ~70x the baseline cost, indicating closure currently depends on configurations that are not practical at scale. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。