测试大模型记忆系统能否跟上世界状态变化,发现现有方法普遍失效。
Can Agent Memory Systems Track Evolving State?

- 设计新基准StateMemBench,评估记忆是否随状态更新而调整。
- 提出StateMem方法,当前状态准确率提升1.8倍,最高达0.363。
- 可作为轻量封装升级现有记忆系统,效果提升32到67点。
随着基于大语言模型的智能体被用于更长、更高风险的任务,其记忆系统仍存在关键缺陷。现有记忆评测多聚焦于回忆类任务,但我们认为有效记忆系统必须能追踪世界状态的动态演变——当事实、约束或决策随长时间交互被修改时,回答应反映最新状态而非过时信息。我们定义此能力为状态追踪,并构建了StateMemBench基准,包含234个多会话场景,覆盖两种对话长度范式。其封闭池评分机制可区分答案是否反映当前状态、过时状态或失败,从设计上分离出状态追踪错误与其他错误。分析表明,现有记忆系统、检索增强基线和长上下文基线均难以应对该任务。为此,我们提出StateMem方法,显式追踪状态替换关系与依赖关系,在DeepSeek-V4-Flash上相较最强同架构基线当前状态准确率从0.205提升至0.363(提升1.8倍),在Qwen-3.5-9B上相较最强记忆系统从0.149提升至0.233(提升1.6倍),同时保持与长上下文基线相当的竞争力。最后,我们证明该状态策略可作为轻量单次调用封装,应用于六种不同记忆与检索后端,使当前状态准确率提升32至67点;长度与成本匹配的对照组表明其中15至32点来自状态结构本身,而非额外上下文。
原文摘要 · Abstract (English)
As LLM-based agents are deployed for longer and higher-stakes tasks, their memory systems continue to have crucial gaps. While existing memory benchmarks focus largely on recall-shaped tasks, we argue an effective memory system must track the evolving state of the world; as facts, constraints, and decisions are revised over a long interaction, answers must reflect the current state and not a superseded one. We define this capability as state tracking and instantiate it in StateMemBench, a benchmark of 234 multi-session scenarios spanning two conversation-length regimes. Its closed-pool grading scores whether an answer reflects the current state, the superseded state, or fails otherwise, separating state-tracking failures from other errors by construction. Our analysis shows that this task is challenging for existing memory systems, retrieval-augmented baselines, and long-context baselines. We then present StateMem, a state-first memory method that explicitly tracks supersession and relational dependencies, and show it improves current-state accuracy over the strongest same-backbone baseline by 1.8x (0.205 -> 0.363) on DeepSeek-V4-Flash and over the strongest memory system by 1.6x (0.149 -> 0.233) on Qwen-3.5-9B, while remaining competitive with the long-context baselines. Finally, we show the same state approach can be applied as a lightweight single-call wrapper over existing memory systems, lifting current-state accuracy by +32 to +67 points on StateMemBench across six memory and retrieval backends. A length- and cost-matched control attributes +15 to +32 of those points to state structure rather than added context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。