评测智能体长期记忆能力,帮助其成为领域专家。
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

- 设计新基准,评估智能体对网页环境经验的长期记忆能力。
- 提出两种记忆方法,其中编码代理方法准确率达72.5%。
- 适合研究长时记忆、智能体协作与个性化环境建模者。
长期记忆对特定网络环境中智能体的成功至关重要,其需记住界面功能、状态变化、工作流程及常见失败模式。然而,现有智能体记忆评测多聚焦用户历史、短轨迹或下游任务成功率,缺乏对记忆系统是否有效内化环境经验的直接评估。为此,我们提出LongMemEval-V2(LME-V2),一个评估记忆系统能否帮助智能体获得成为领域专家所需经验的基准。LME-V2包含451个手工构建的问题,覆盖五项核心记忆能力:静态状态回忆、动态状态追踪、工作流程知识、环境陷阱认知和前提意识。问题配以最长达500条轨迹、总计115M token的历史数据。采用上下文收集范式:记忆系统读取历史轨迹并返回紧凑证据用于问答。我们提出两种方法:AgentRunbook-R,基于RAG的高效记忆,使用原始状态、事件和策略笔记的知识池;AgentRunbook-C,将轨迹存储为文件,并调用编码代理在增强沙箱中提取证据。实验表明,AgentRunbook-C平均准确率达72.5%,优于最强RAG基线(48.5%)和现成编码代理基线(69.3%)。尽管性能提升显著,编码代理方法仍存在高延迟问题。虽已拓展准确率-延迟权衡边界,仍有改进空间。这些结果确立了LME-V2作为评估环境经验长期记忆系统的挑战性平台。
原文摘要 · Abstract (English)
Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments. LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. Questions are paired with history trajectories containing up to 500 trajectories and 115M tokens. We use a context gathering formulation: memory systems consume history trajectories and return compact evidence for downstream question answering. We propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show that AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and the off-the-shelf coding agent baseline (69.3%). Despite the strong performance gains, coding agent based methods have high latency costs. While AgentRunbook-C advances the accuracy-latency Pareto frontier, substantial room for improvement remains. Together, these results establish LME-V2 as a challenging testbed for developing long-term memory systems for environment experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。