首个系统性评测对话智能体长期记忆策略的基准，揭示外存键值存储最优但代价高。
AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents
- 统一测试五种记忆管理方法，相同数据与评估标准。
- 外存键值存储在长程召回上遥遥领先，其他方法几乎失效。
- 适合研究长期记忆、评估系统可靠性的开发者与研究人员。
长期记忆仍是对话型AI智能体的关键瓶颈，其有限的上下文窗口无法支持数千轮对话中的连贯回溯。本文提出AgentMemBench，一个统一且可复现的基准，评估五种记忆管理策略：上下文窗口（ICW）、外部键值存储（EKV）、基于图的事件记忆（GEM）、压缩摘要（CBS）和网络增强记忆（WAM）。所有方法在三个公开数据集上进行测试：长程多会话对话（LoCoMo）、任务导向文档对齐（MultiDoc2Dial）和人格驱动多会话聊天（MSC），使用召回率@k、MRR、nDCG@k、答案F1、LLM判断的忠实度分数、内存占用和延迟等指标，覆盖491个标注问题回合。生成与评判均采用Qwen2.5-7B-Instruct（4-bit），贪婪解码以保证确定性。结果表明：（1）EKV在各项质量指标上全面领先（宏召回率@5为0.792，MRR为0.677，F1为0.156，忠实度为0.354）；（2）长程召回至关重要：在LoCoMo中，金标回合跨多个会话，其余方法召回率≤0.005，仅EKV达到0.573，说明窗口、摘要与实体图在长距离下失效，唯有密集检索有效；（3）CBS为第二优，召回率为0.556；（4）WAM与ICW在语料内召回相当，因外部结果无语料溯源；（5）EKV的召回优势伴随显著内存开销（约5,100对比约300令牌），体现精度与效率的明确权衡。此外，我们还用同一评估框架测试了两个已发表系统（MemGPT/Letta、HippoRAG），并公开全部代码、环境与结果资产以实现完全可复现。
原文摘要 · Abstract (English)
Long-term memory remains a critical bottleneck for conversational AI agents, whose finite context windows cannot support coherent recall across thousands of turns. We present AgentMemBench, a unified, reproducible benchmark evaluating five memory management strategies under identical conditions: in-context windowing (ICW), external key-value store (EKV), graph-based episodic memory (GEM), compression-based summarisation (CBS), and web-augmented memory (WAM). All are assessed across three public datasets covering long-term multi-session dialogue (LoCoMo), task-oriented document grounding (MultiDoc2Dial), and persona-grounded multi-session chat (MSC), using Recall@k, MRR, nDCG@k, Answer F1, an LLM-judge Faithfulness score, Memory Footprint, and Latency over 491 annotated question turns. Generation and judging both use Qwen2.5-7B-Instruct (4-bit), with greedy decoding for determinism. Our results show that (1) EKV dominates on every quality axis (macro Recall@5 0.792, MRR 0.677, F1 0.156, Faithfulness 0.354); (2) long-range recall is decisive: on LoCoMo, where the gold turn lies many sessions back, ICW, WAM, GEM, and CBS retrieve almost nothing (Recall@5 <= 0.005) while EKV alone reaches 0.573, showing that recency windows, summaries, and entity graphs collapse at long horizons and only dense retrieval scales; (3) CBS is the runner-up on retrieval (0.556); (4) WAM equals ICW on in-corpus recall by construction, since external results carry no in-corpus provenance; and (5) EKV's recall advantage carries a footprint cost (~5,100 vs ~300 tokens for ICW/WAM), an explicit accuracy-efficiency trade-off. We additionally evaluate two published memory systems (MemGPT/Letta, HippoRAG) against the same harness, and release all code, environment, and result artefacts for full reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。