评测大模型长期对话记忆，发现图结构记忆更擅长跨空间推理。
EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval
- 用图结构存储对话信息，提升跨会话关联能力。
- 在跨空间推理任务上得分0.6532，优于全上下文提示的0.6291。
- 适合研究长期记忆系统与多任务优化矛盾的开发者。
大型语言模型助手需在多轮会话中保持并推理累积信息。我们提出EngramaBench，一个基于五位角色、一百个跨会话对话及一百五十个查询的基准测试，涵盖事实回忆、跨空间整合、时间推理、对抗性回避和涌现合成。评估了Engrama（图结构记忆系统）与GPT-4o全上下文提示、Mem0（开源向量检索系统）的表现。三者均使用GPT-4o作为回答模型，仅比较记忆架构差异。GPT-4o全上下文获最高综合分0.6186，Engrama全局得分为0.5367，但在跨空间推理上以0.6532高于全上下文提示的0.6291（n=30）。Mem0成本最低但性能显著更弱（0.4809）。消融实验显示，提升跨空间推理的组件会降低整体综合得分，揭示结构化记忆专精与全局优化间的系统级权衡。
原文摘要 · Abstract (English)
Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce EngramaBench, a benchmark for long-term conversational memory built around five personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall, cross-space integration, temporal reasoning, adversarial abstention, and emergent synthesis. We evaluate Engrama, a graph-structured memory system, against GPT-4o full-context prompting and Mem0, an open-source vector-retrieval memory system. All three use the same answering model (GPT-4o), isolating the effect of memory architecture. GPT-4o full-context achieves the highest composite score (0.6186), while Engrama scores 0.5367 globally but is the only system to score higher than full-context prompting on cross-space reasoning (0.6532 vs. 0.6291, n=30). Mem0 is cheapest but substantially weaker (0.4809). Ablations reveal that the components driving Engrama's cross-space advantage trade off against global composite score, exposing a systems-level tension between structured memory specialization and aggregate optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。