解决AI Agent知识过时问题,让检索系统自动淘汰失效信息
Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge
- 用双时间戳账本记录事实变更,自动淘汰过时信息
- 在动态知识任务中准确率达0.95-1.00,远超RAG的0.20-0.47
- 无需大模型重排,推理延迟仅2.1秒,适合实时应用
检索增强生成(RAG)使智能体可访问累积知识,但缺乏时间模型。当事实更新(如函数重命名或API重构)时,旧值与新值的嵌入相似度几乎相同,导致智能体要么回避回答,要么提供过时信息。我们发现这是结构性缺陷:在校准数据集上,余弦相似度区分矛盾事实与重复事实的AUROC仅为0.59(接近随机)。为此提出MemStrata,一种维护时间有效性的检索记忆系统。它像RAG一样存储事实以保持静态召回,但当事实被推翻时,通过确定性(主语, 关系, 宾语)取代规则,在双时间戳账本中退役旧值,无需相似度阈值或大模型调用。在本地部署的7B模型上,六项基准测试显示,MemStrata在静态知识上与RAG持平,在动态知识上准确率达0.95-1.00(而RAG仅0.20-0.47)。核心结果为:要求作答时,RAG有15-40%概率输出过时内容;而MemStrata将其降至约0%——这是RAG无法规避的失败类别。其检索延迟仅约2.1秒,远低于大模型重排基线的16-18秒。我们发布了评估框架、数据集及无标记评价协议,用于知识演进下的记忆评估。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) gives agents access to accumulated knowledge, but has no model of time. When a fact changes (e.g., a function is renamed or API restructured), RAG retrieves both the stale and current value with near-identical embedding similarity. The agent then either abstains or serves the superseded fact. We show this is a structural problem: on a calibrated dataset, cosine similarity distinguishes a contradicted fact from a duplicated one with AUROC 0.59 (near chance), as contradictions are often more embedding-similar to the original than rephrased duplicates. We present MemStrata, a retrieval memory maintaining temporal validity. It stores facts like RAG, preserving static recall, but when a fact's value is contradicted, a deterministic (subject, relation, object) supersession rule retires the stale value in a bi-temporal ledger - with no similarity threshold and no LLM call. Across six benchmarks run locally with a 7B model, MemStrata ties RAG on static knowledge and reaches 0.95-1.00 accuracy on evolving knowledge (where RAG reaches 0.20-0.47). The central result is the stale-fact-error rate: when required to answer, RAG serves superseded values 15-40% of the time; MemStrata drives this to ~0%, a failure class RAG cannot avoid. MemStrata achieves this at retrieval latency (~2.1s) versus ~16-18s for LLM-reranking baselines. We release the harness, datasets, and a marker-free evaluation protocol for memory under knowledge evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。