arXiv:2605.06527cs.CL2026-05被引 20

测试大模型何时该更新过时记忆,发现多数模型仍会误用旧信息。

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

论文配图:STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
图 1 · 摘自论文原文
  • 设计三维度评测框架,检测记忆是否过时、拒绝错误前提、主动适应新状态
  • 400个真实场景测试显示顶尖模型准确率仅55.2%,普遍忽略隐含冲突
  • 适合关注智能体长期记忆与推理一致性的研究者和开发者

大型语言模型代理日益需要维护连贯的长期个性化记忆,但现有基准主要评估静态事实检索,忽视了在新证据出现时修正存储信念的能力。我们识别出一个关键且未被充分探索的缺陷:隐含冲突——后续观察否定早期记忆,但无显式否定,需上下文推断与常识推理才能察觉。为此,我们提出STALE基准,包含400个专家验证的冲突场景(共1,200个评估查询),覆盖100多个日常话题,上下文长度达150K tokens。我们提出三维探测框架,测试状态修正(检测旧信念已过时)、前提抵抗(拒绝虚假预设过时状态的提问)、隐含策略适应(在下游行为中主动应用更新后的状态)。对前沿大模型与专用记忆框架的系统评估揭示:从获取更新证据到实际应用之间存在普遍差距,即使最优模型整体准确率也仅为55.2%。模型常接受用户问题中嵌入的过时假设,并难以识别某一状态变化应如何使相关记忆失效。为建立状态感知记忆的初始基线,我们进一步提出CUPMem原型,通过结构化状态整合与传播感知搜索强化写入时的修订能力,表明显式状态裁决是构建稳健代理记忆的有前景方向。

原文摘要 · Abstract (English)

Large Language Model (LLM) agents are increasingly expected to maintain coherent, long-term personalized memory, yet current benchmarks primarily measure static fact retrieval, overlooking the ability to revise stored beliefs when new evidence emerges. We identify a critical and underexplored failure mode, Implicit Conflict: a later observation invalidates an earlier memory without explicit negation, requiring contextual inference and commonsense reasoning to detect. To rigorously evaluate this capability, we introduce STALE, a benchmark of 400 expert-validated conflict scenarios (1,200 evaluation queries across three probing dimensions) spanning over 100 everyday topics with contexts up to 150K tokens. We propose a three-dimensional probing framework that tests State Resolution (detecting that a prior belief is outdated), Premise Resistance (rejecting queries that falsely presuppose a stale state), and Implicit Policy Adaptation (proactively applying updated states in downstream behavior). A systematic evaluation of frontier LLMs and specialized memory frameworks reveals a pervasive gap between retrieving updated evidence and acting on it, with even the best evaluated model achieving only 55.2% overall accuracy. Models often accept outdated assumptions embedded in a user's query, and they struggle to recognize when a change in one aspect of the user's state should invalidate related memories. To establish an initial baseline for state-aware memory, we further present CUPMem, a prototype that strengthens write-time revision through structured state consolidation and propagation-aware search, suggesting that explicit state adjudication is a promising direction for robust agentic memory.

大模型代理记忆更新推理一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。