arXiv:2606.24595cs.CL2026-06被引 2

首次通过还原用户状态评估大模型长时记忆,发现记忆能力与任务完成度无关。

MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery

论文配图:MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery
图 1 · 摘自论文原文
  • 设计新基准MEMPROBE,从代理留下的记忆中逆向重建用户状态
  • 50名模拟用户共1550个隐藏维度,真实状态可被精准还原
  • 顶尖记忆系统恢复率仅0.6,说明记忆质量需单独评估

长时记忆能提升大模型代理在多轮交互中的理解能力,但当前评估主要依赖下游行为(如回答准确率、个性化程度),间接测试记忆效果,难以直接审计记忆本身。本文提出将长时记忆视为可审计的后交互产物:在常规服务后,能否从代理留下的记忆中重构出用户的结构化状态?为此构建了MEMPROBE基准,让配备记忆的代理协助50名模拟用户(每人含31个隐含维度,共1550个恢复目标),每个用户携带基于分类体系的隐藏状态库,在可控泄露的任务轨迹后,通过全量访问与顶k检索两种方式重建其状态。基于合成真值实现高效可扩展测量,验证了5种代表性记忆系统。实验发现,任务完成率接近饱和(甚至无记忆基线也达标),而类别均衡恢复率仅为约0.6,且顶k检索下进一步下降。MEMPROBE是首个直接研究记忆恢复的基准,可直接重建并评分系统留存的用户状态,为未来记忆代理提供明确优化目标。

原文摘要 · Abstract (English)

Long-term memory promises LLM agents that grow more capable across sessions, maintaining an accurate, evolving understanding of the user that interaction forms. In practice, however, this memory is evaluated mostly through downstream behavior, such as later answers, personalization quality, or task success, which tests that understanding only indirectly and leaves the memory artifact itself largely unaudited. We argue that long-term memory should instead be evaluated as an auditable post-interaction artifact: after ordinary assistance, what structured user state can be reconstructed from the memory the agent leaves behind? We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users, each carrying a hidden, taxonomy-anchored user-state bank, across a trajectory of leak-controlled tasks, after which that bank is reconstructed from the agent's resulting memory under both full-store and top-k access. Built on synthetic ground truth for efficient, scalable measurement, MEMPROBE spans 50 simulated users with 31 hidden dimensions each (1,550 recovery targets) and tests 5 representative memory systems. Testing state-of-the-art memory agents, we find that successful assistance and recoverable memory behave as distinct capabilities. Task completion nearly saturates, even for a memoryless baseline, while category-balanced recovery stays moderate (about 0.6) and drops further under top-k retrieval. MEMPROBE is the first benchmark to study memory recovery directly, reconstructing the user state a system retains and scoring it against ground truth. We see recovery as a concrete objective for future memory agents to optimize, and MEMPROBE as a step toward an environment where agents are trained to remember their users, growing more faithful the longer they know them.

长时记忆评估基准用户建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。