评估大模型工具代理记忆的实际价值，发现记忆能显著提升任务成功率，但成本高昂且实现方式影响巨大。
When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents
- 设计新基准MERIT，量化记忆操作的代价与收益，真实模拟长期记忆在任务中的作用。
- 有记忆时任务成功率达0.55-1.00，但更新后信息检索失败率高，仅55%正确使用。
- 结构化事实存储优于混合方案，全量回放虽有效但不经济，性价比仅为最优方案的1/3。
当前对大模型代理长时记忆的评估依赖对话回忆基准（LoCoMo、LongMemEval），仅衡量基于对话历史的问答能力，未检验记忆是否真正改变工具使用行为。本文提出MERIT（真实仪器化任务的记忆评估），一个包含成本核算的基准与工具链，用于评估记忆在任务执行中的边际效用。MERIT涵盖三个领域中依赖前期事实的片段式工具任务，通过自动化泄漏检测验证依赖性；设置难度阶梯直至更新后事实回忆；引入可控记忆污染；并全程计量每个记忆操作的令牌与美元开销。在23,440次评分实验（总花费42.57美元）中，两代原型测试（gpt-4.1-mini）及预注册三模型×三种子网格（GPT-4.1、Claude Haiku 4.5；记忆侧固定），记忆使依赖任务成功率从经泄漏验证的0.00提升至0.55-1.00。对于更新事实，嵌入检索表现不稳定（0.30-0.95，最大种子差异0.45），而代理仅55%时间正确使用检索到的信息；更新写入存储（结构化事实库及显著的LLM摘要）维持在0.70-1.00；混合方案反而劣于纯事实库。最新一代探测（Claude Sonnet 5，基于干净全回放控制）复现该模式。更换记忆实现方式可使任务成功率变动高达60点，全量回放从未具备经济性：最优条件每美元边际效用为其他方案的2.7-3.9倍。相关基准、工具链与全部数据轨迹已公开。
原文摘要 · Abstract (English)
Long-term memory for LLM agents is evaluated today by conversational recall benchmarks (LoCoMo, LongMemEval), which measure question answering over dialogue history, not whether remembered facts change what a tool-using agent does. We present MERIT (Memory Evaluation for Realistic Instrumented Tasks), a benchmark and harness that measures the marginal utility of memory for task-executing agents under explicit cost accounting. MERIT provides episodic tool-use tasks in three domains whose dependence on earlier-episode facts is verified by an automated leak check; a difficulty ladder ending in updated-fact recall; controlled memory corruption; and full token and dollar metering of every memory operation. Across 23,440 scored episodes ($42.57), a two-generation pilot on gpt-4.1-mini and a preregistered 3-model x 3-seed grid (GPT-4.1, Claude Haiku 4.5; memory side held fixed), memory lifts dependent-task success from a leak-verified floor of 0.00 to 0.55-1.00. On updated facts, embedding retrieval collapses unpredictably (0.30-0.95 across models; max seed gap 0.45), and agents act on a correctly retrieved value only 55% of the time, while update-on-write stores (a structured fact store and, notably, LLM summarization) remain at 0.70-1.00; the hybrid is worse than the fact store alone. A latest-generation spot-check (Claude Sonnet 5, gated on a clean full-replay control) reproduces the pattern. Swapping a memory's implementation moves task success by up to 60 points, and full replay is never economical: the best condition per domain delivers 2.7-3.9x its marginal utility per dollar. We release the benchmark, harness, and all traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。