对比了三种记忆系统在长对话中的服务成本，发现成本不可预测且难胜全量重传。
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
- 通过实测比较三种记忆系统与两种基准策略的成本与效果
- 记忆系统成本比预估高18%-69%，受内部行为影响大
- 无系统同时胜出准确率和成本，选型需权衡
长对话智能体依赖记忆系统避免重复发送完整对话，但其服务成本缺乏系统性评估。本文在两个模型骨干下，对三种记忆系统（Mem0、Hindsight、Mastra 观察记忆）与两种基准策略（固定大小滚动窗口、重传完整对话）进行了对比，涵盖最长400轮的对话，并在665个LoCoMo问题上同步测量回答准确率。结果表明：记忆系统的服务成本无法仅由对话长度和消息大小预测，回归模型对基准策略拟合良好，但对记忆系统误差达18%-69%，实际成本主要取决于内部记忆行为；进一步的盈亏分析显示，记忆系统何时比重传完整对话更便宜，高度依赖具体系统与模型骨干，从第十几轮即生效到400轮内均未实现；此外，各系统准确率在21%-54%之间波动，模型骨干对成本的影响与记忆系统相当。
原文摘要 · Abstract (English)
Long-running conversational agents increasingly rely on a memory system to avoid resending the whole conversation each turn, yet how much that costs to serve has received little systematic benchmarking. We compare three memory systems (Mem0, Hindsight, and Mastra Observational Memory) against two reference strategies -- a fixed-size rolling window and resubmitting the full transcript -- across two backbones and conversations of up to 400 turns, pairing every cost measurement with answer accuracy on 665 LoCoMo questions. First, a memory system's serving cost cannot be predicted from conversation length and message size alone: a regression that tracks the two reference strategies closely misses the memory systems by 18-69%, their cost driven instead by internal memory behavior. Second, a break-even analysis shows that whether -- and when -- a memory system becomes cheaper to serve than the full transcript is highly sensitive to the system and the backbone, from the first tens of turns for the cheapest to never within 400 turns for the most expensive. Third, no system wins on both axes: accuracy spans 21-54%, and the backbone choice drives cost as much as the memory system does.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。