评估记忆系统时,展示方式会掩盖真实机制效果。
Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation

- 区分机制与展示效果,控制呈现方式变量
- 细粒度修订记录仅提升0.025表现,主要来自展示优势
- 粗粒度失效机制在实际查询中更有效,适合多数场景
AI系统越来越多地从自我更新的记录中检索信息:问题线程、百科历史、政策日志和长对话。挑战不仅在于找到相关证据,还在于判断哪些主张仍有效、哪些已被取代,以及何时应放弃回答。结构化记忆通过类型化边、时间更新和冲突状态试图解决此问题,但现有评估常同时改变机制与提示展示方式。我们提出证据状态修订(Evidence-State Revision)任务,在2,907个高一致性的问答上对比平铺检索、粗粒度边失效与细粒度RevisionLedger方法。一个渲染匹配的对照组(相同布局,禁用废弃)揭示核心混淆:当值被修改后又恢复时,RevisionLedger看似比平铺基线提升+0.182,但几乎全部收益来自展示优化;其机制残差接近零(+0.021至+0.025,跨两组评审者)。控制展示后,粗粒度失效是唯一对当前状态查询有实际提升的方法,优于细粒度记录0.084;同样的查询充分性原则表明,溯源只需保留已失效证据,无需更丰富的类型标注。记忆评估应固定渲染方式,去淘汰系统的部署应选择覆盖查询所需的最简保留状态。
原文摘要 · Abstract (English)
AI systems increasingly retrieve from records that revise themselves: issue threads, encyclopedic histories, policy logs, and long conversations. The challenge is not only finding relevant evidence, but deciding which claims remain in force, which were superseded, and when to abstain. Structured memories promise to solve this with typed edges, temporal updates, and conflict status, yet evaluations often change mechanism and prompt presentation together. We study this as Evidence-State Revision, comparing flat retrieval, coarse edge invalidation, and fine-grained RevisionLedger on 2,907 high-agreement questions from GitHub, multi-repo issue histories, Wikipedia, and DyKnow-style temporal streams. A render-matched control (same layout, deprecation disabled) reveals the central confound: when a value is changed and later restored, RevisionLedger appears to beat a flat baseline by +0.182, but almost all the gain comes from easier presentation; the fine-grained mechanism residual is indistinguishable from zero (+0.021 to +0.025 across two judge families). After presentation is controlled, coarse invalidation is the only mechanism that pays for current-state queries, beating the fine ledger by 0.084; the same query-sufficiency principle says provenance mainly needs retained invalidated evidence, not richer typing. Memory evaluations should hold render fixed, and deprecation-aware systems should deploy the coarsest retained state that covers their queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。