对比记忆写入与检索对大模型代理性能的影响，发现检索方式比写入策略更重要。
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory
- 设计诊断框架，测试三种写入策略与三种检索方法的组合效果。
- 检索方法差异导致准确率相差20个百分点（57.1%至77.2%），远超写入策略影响。
- 原始片段存储表现优于复杂提取/摘要，说明当前检索机制难以弥补信息丢失。
具备记忆能力的大语言模型代理通过存储和检索历史交互信息来增强决策能力，但记忆的写入方式与检索方式的相对重要性尚不明确。本文提出一种诊断框架，分析不同写入策略、检索方法及记忆利用行为对性能的影响，并在LoCoMo数据集上开展3×3实验，对比三种写入策略（原始片段、Mem0式事实提取、MemGPT式摘要）与三种检索方法（余弦相似度、BM25、混合重排序）的效果。结果表明，检索方法是主导因素：平均准确率在不同检索方法间波动达20个百分点（57.1%至77.2%），而写入策略仅造成3-8个百分点差异。原始片段存储无需额外LLM调用，其表现可媲美甚至优于代价高昂的有损方法，暗示现有记忆流程可能丢弃了下游检索无法补偿的有效上下文。失败分析显示，性能下降主要发生在检索阶段而非使用阶段。因此，在当前检索实践下，提升检索质量带来的收益大于提升写入复杂度。
原文摘要 · Abstract (English)
Memory-augmented LLM agents store and retrieve information from prior interactions, yet the relative importance of how memories are written versus how they are retrieved remains unclear. We introduce a diagnostic framework that analyzes how performance differences manifest across write strategies, retrieval methods, and memory utilization behavior, and apply it to a 3x3 study crossing three write strategies (raw chunks, Mem0-style fact extraction, MemGPT-style summarization) with three retrieval methods (cosine, BM25, hybrid reranking). On LoCoMo, retrieval method is the dominant factor: average accuracy spans 20 points across retrieval methods (57.1% to 77.2%) but only 3-8 points across write strategies. Raw chunked storage, which requires zero LLM calls, matches or outperforms expensive lossy alternatives, suggesting that current memory pipelines may discard useful context that downstream retrieval mechanisms fail to compensate for. Failure analysis shows that performance breakdowns most often manifest at the retrieval stage rather than at utilization. We argue that, under current retrieval practices, improving retrieval quality yields larger gains than increasing write-time sophistication. Code is publicly available at https://github.com/boqiny/memory-probe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。