用剧本生成真实记忆数据,发现长时记忆系统排名会反转。
Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
- 先生成带有效期和来源的事件剧本,再渲染对话,确保答案真实可验。
- 9周后内存系统排名逆转:图结构优于缓存映射,召回率从72%升至90%。
- 适合研究长期记忆、模型可信度或想复现基准的开发者。
现有大模型代理记忆评测通常先生成对话再提取答案,存在标签错误和污染问题,且多关注短时交互。本文反向设计:先由种子人生剧本采样器生成带有效期、波动等级和信源的事件事实;再由大模型渲染聊天与邮件内容;最后通过保真度验证器逐项确认所有预植事实。问题由剧本机械生成,保证黄金答案与剧本一致,并独立验证可回答性。合成虚构语料库包含约380个问题、15种类型,引入了事实有效期、发送/接收信任区分、良性环境下的注入探测及按时间点的问题集等新特性。在五种记忆架构与无记忆对照组(固定回答者、版本化大模型评判、三次重复、两个时间跨度)上测试,发现后台排名随历史长度反转:预算受限的精选映射记忆在三周领先,九周后被驱逐内容召回率从96%降至72%,而基于出处类型的图结构上升至90%;在六位用户完全跨家族重评下,该反转显著(精确p=0.031)。完整历史基线在短期表现不输最佳记忆系统,但九周后无独立优势,读取成本约为两倍。写作阶段质量与下游表现强相关(弱写事实失败率24%对2%),注入抗性取决于出处边界是否保留。层级架构在两种场景下均最优(短期96.8%),其代码已开源为Veracium库,含语料生成器与评估框架。
原文摘要 · Abstract (English)
Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We invert the pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction and separately validated for answerability. The synthetic, fictionalized corpus (~380 questions, 15 types) embeds features absent from the benchmarks we survey: per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets. Benchmarking five memory architectures against a no-memory control (fixed answerer, versioned LLM judge, three replicates, two horizons), we find backend rankings invert with history length: the budgeted curated-map memory that leads at three weeks loses recall of evicted content by nine weeks (96% to 72%) while a provenance-typed graph rises to 90%; the inversion is positive for all six users under complete cross-family re-judging (exact p=0.031). A full-rendered-history baseline ties or exceeds the best memory system at the short horizon but shows no judge-independent advantage at nine weeks, at about twice the read cost. Write-stage quality strongly correlates with downstream quality (weakly-written facts fail 24% vs 2%), and injection resistance tracked whether provenance boundaries survive representation. A layered architecture performs best among the memory systems in both regimes (96.8% short-horizon) and is released as Veracium, an open-source library, with the corpus generator and harness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。