arXiv:2605.11325cs.IRcs.AI2026-05被引 1

提出首个关注检索精度的基准,解决大模型记忆检索不准问题

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

  • 设计结构化信念状态,先于推理完成检索与作用域解析
  • 新基准测试显示基线系统精确率仅0.22,而新方法全任务达完美检索
  • 适合研究大模型记忆机制、需高精度信息召回的场景

现有大模型记忆评估仅关注答案质量,导致系统可通过倾倒全部信念库实现高召回率,掩盖严重精度缺陷。我们在多个嵌入模型上发现,基于相似度的领域语料检索难以区分语义相近的信念。多轮话题漂移进一步加剧检索噪声,提升延迟与成本。为此,我们提出PrecisionMemBench基准(89个案例),测量精确率、噪声隔离、会话延迟和信念可变性。同时引入Tenure结构化信念存储代理,在模型处理提示前完成检索与作用域解析,并注入带类型标记的信念状态作为上下文指令,消除模型对是否调用记忆的自主判断。13种配置评估显示,Tenure在所有活跃、非会话及会话测试中均实现100%正确检索通过;而基线系统活跃通过率未达一半,精确率集中于0.22以下。结果表明,当前记忆系统虽能存信息,却无法干净地检索;这一结构性缺陷被传统答案质量评估所掩盖。

原文摘要 · Abstract (English)

Current LLM memory benchmarks evaluate answer quality rather than retrieval accuracy. Consequently, a system that dumps its entire belief store can achieve perfect recall and mask severe precision failures. We show this evaluation gap persists across multiple embedding models where similarity-based retrieval over domain-specific corpora inherently struggles to isolate target beliefs from semantically proximate ones. Furthermore, multi-turn topic drift compounds this retrieval noise while driving up latency and operational costs. To decouple retrieval quality from generative performance, we introduce PrecisionMemBench, an 89-case benchmark measuring precision, noise isolation, session latency, and belief mutability. We also present Tenure, a structured belief-store proxy that resolves scope and retrieval before inference and injects typed belief state as ambient instruction before the model sees the prompt, removing model-side discretion over whether memory is consulted. Evaluated across 13 configurations, Tenure achieves perfect retrieval passes across all active, non-session, and session test cases. In contrast, the baseline configurations fail to reach even half of the active passes, with precision scores clustering at 0.22 and below. Our results demonstrate that while current memory systems successfully store information, they fail to retrieve it cleanly; a structural vulnerability that traditional answer-quality benchmarks conceal.

大模型记忆检索精度信念状态基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。