用知识点维度评估大模型长期记忆，发现现有方法忽略证据使用问题。
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
- 以用户事实为单位，控制时间、问题类型和证据条件进行测试
- 10倍以上失败时证据仍可被检索，说明瓶颈在证据利用而非获取
- 适合关注长期记忆系统设计的研究者与开发者
大型语言模型代理越来越多地跨会话维护用户长期记忆。然而，当前评估通常基于对问题行或会话的准确率汇总，因独立评分无法反映同一事实在不同条件下的表现。本文提出MemTrace，以单一用户事实为测量单元，从记忆年龄（会话间隔）、问题类型（当前状态、历史状态、变化轨迹）和证据条件（存在、缺失、矛盾前提）三个维度进行控制测试。在四个范式中评估13种记忆配置，发现聚合准确率掩盖了关键差异：恢复当前与历史状态并不等于追踪变化过程，安全回避也不代表修正错误前提。主要瓶颈在于证据使用而非检索——系统失败时，证据可被检索的频率是缺失的10倍。结果表明，提升长期记忆需优化已获证据的利用能力，而非单纯增加存储或检索能力。
原文摘要 · Abstract (English)
LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change. We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings. Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact's current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing. These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。