arXiv:2607.12893cs.AIcs.CL2026-07被引 3

给长对话记忆设计操作级评测,揭示模型记忆失效的深层原因。

MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations

论文配图:MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
图 1 · 摘自论文原文
  • 将记忆视为可追踪的操作生命周期,而非静态事实存储
  • 在长上下文场景下,现有模型重建记忆轨迹能力仍很弱
  • 提供六类操作级探针,适合评估记忆系统的可靠性与可解释性

长期记忆已成为支持用户跨多轮、多会话交互的大型语言模型代理的核心能力。然而,现有基准主要通过下游问答任务评估记忆,仅关注最终答案正确性,未能区分记忆失败的多种成因,如遗漏关键事实引入、错误绑定操作目标或依赖已更正后的过时值。这可能导致系统因不一致或不安全的记忆状态而获得正确答案的误判。本文提出,动态长周期交互中记忆并非静态事实集合,而是一系列显式操作的生命周期,包括记忆、遗忘、更新、反思及其组合。我们构建了MemOps基准,将对话记忆重构为操作序列,并以结构化轨迹记录每条记忆事件的触发、目标、作用范围、状态变迁及支持证据。通过可控生成流程,将这些操作嵌入长任务型对话,生成黄金操作轨迹及六类操作级探针,在相邻证据与长上下文两种设置下进行评估。结果表明,当前系统在长上下文检索、参数化与管理内存系统中仍存在显著差异,会话级检索优于回合级,长上下文模型在重建有序记忆状态轨迹方面表现明显不足。该研究推动长期记忆评估从最终答案评分转向可解释的操作级诊断。

原文摘要 · Abstract (English)

Long-term memory has become a foundational capability for LLM-based agents that accompany users across extended, multi-session interactions. Existing benchmarks, however, evaluate such memory almost exclusively through downstream question answering, scoring only the correctness of a final answer. This black-box formulation conflates the heterogeneous causes of memory failure, such as missing the introduction of a relevant fact, binding an operation to the wrong target, or relying on stale values after a correction. As a result, it can credit correct answers despite their reliance on inconsistent or unsafe memory states. In this paper, we argue that, in dynamic long-horizon interactions, memory is not a static collection of facts but a lifecycle of explicit operations, including remembering, forgetting, updating, reflecting, and their compositions. We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations and represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence. A controllable generation pipeline embeds these operations into long, task-oriented conversations and produces gold operation traces together with six categories of operation-level probes, evaluated under both adjacent-evidence and long-context settings. Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals, revealing that current systems remain far from uniformly reliable. For instance, session-level retrieval outperforms turn-level retrieval, and long-context models remain notably weak at reconstructing ordered memory-state trajectories. These results move long-term memory evaluation from final-answer scoring toward interpretable, operation-level diagnosis.

长期记忆评估基准对话系统可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。