提出可控评估框架，揭示记忆系统评价中的隐藏干扰因素
MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation
- 逐项控制变量，在长对话评测集上隔离内存方法、模型、检索等影响
- 同一管道下更换嵌入模型可导致准确率波动6.2个百分点，结论可能反转
- 发现自记忆性能低于基础检索，部分高成本方案仅在特定场景有效
智能体记忆系统常通过RAG和全上下文基准进行评估，但报告的提升常混杂语言模型、嵌入模型或检索流程的变化，难以判断真实贡献。我们提出MemDelta，一种受控评估协议，在LongMemEval-S（500个问题，50+会话，三种模型家族）上逐项变化单一组件。四项发现：(1) 语义重复的RAG与GPT-4o-mini全上下文表现接近（47.2% vs. 49.8%，p=0.34），但跨模型趋势反转：Gemini从全上下文获益+14pp，Sonnet则从RAG获益+31pp，部分因后者拒绝63%全上下文查询；(2) 在相同管道中仅更换嵌入模型，准确率在n=500时波动+6.2pp（p=0.004），Mem0比MiniLM-RAG高+11pp，却低于云RAG 1.2pp，单一变量即可改变结论；(3) 智能体自记忆性能（42%）低于基础检索（47%）；(4) 在6类问题中的2类（n=88），Mem0达到云RAG水平（72.7% vs. 73.9%，p=1.0），但成本高出50倍，显示收益局限而非普适。建议评估时固定嵌入模型，按模型族分层，并在归因前报告写路径开销。
原文摘要 · Abstract (English)
Agent memory systems are increasingly evaluated against RAG and full-context baselines, but reported gains often mix changes in the memory method with changes in the language model, embedding model, or retrieval pipeline, making it unclear what is actually being measured. We present MemDelta, a controlled evaluation protocol that varies one component at a time on LongMemEval-S (500 questions, 50+ sessions, three model families). Four findings emerge: (1) verbatim RAG matches full-context GPT-4o-mini (47.2% vs. 49.8%, p = 0.34), but the ranking reverses across models: Gemini gains +14pp from full context, while Sonnet gains +31pp from RAG, partly because it refuses 63% of full-context queries; (2) swapping only the embedding model in an identical pipeline shifts accuracy by +6.2pp at n = 500 (p = 0.004), and Mem0 beats MiniLM-RAG by +11pp but loses to cloud-RAG by 1.2pp, so one variable flips the conclusion; (3) agent self-memory (42%) underperforms basic retrieval (47%); (4) on 2 of 6 question types (n = 88), Mem0 matches cloud RAG (72.7% vs. 73.9%, p = 1.0) at 50x the cost, suggesting narrow rather than general gains. We recommend memory evaluations fix embedding models across comparisons, stratify by model family, and report write-path cost before attributing gains to architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。