分离证据提取与答案生成,显著提升大模型记忆系统的可靠性。
Reliable Post-Retrieval Assembly for Agent Memory: Separating Evidence Extraction from Policy Execution
- 将证据提取和答案策略执行分步处理,避免多任务混杂导致错误。
- 单跳准确率最高达93%,多跳准确率提升至41%,超越现有最佳结果。
- 适合需要高可靠性的智能体记忆系统研发者参考。
基于大语言模型的记忆系统在检索相关证据后仍会失败,原因在于答案生成阶段同时处理语义筛选、冲突解决、先验抑制和输出生成,形成耦合问题。本文提出后检索组装的结构化接口:先由LLM提取匹配证据,再独立执行答案策略。在MemoryAgentBench v3 FactConsolidation测试中,该流水线在262K规模下实现82%/93%单跳准确率和27%/41%多跳准确率,显著优于所有已报告结果。全链路对照实验显示,仅改变架构即提升平均10.8个百分点(最高21个百分点)。针对性对比表明,仅替换策略执行器贡献平均2.0个百分点,说明主要收益来自任务解耦而非新鲜度机制本身。长时记忆评估未发现整体优势(26/45 对比 29/45;配对精确McNemar p=0.45),表明效果限于带显式版本元数据的当前值问题。
原文摘要 · Abstract (English)
LLM-based memory systems can retrieve relevant evidence yet still fail when answer generation entangles semantic filtering, conflict resolution, prior suppression, and output generation in one step. We study this failure as a problem of post-retrieval assembly. In the MemoryAgentBench (MAB) release used here, FactConsolidation explicitly states that newer facts have larger serial numbers, yet the best reported retrieval/memory result is 54% single-hop and all 22 reported systems score at most 7% multi-hop. We evaluate a structured assembly interface in which an LLM first extracts semantically matching evidence into a candidate representation and a separate stage executes the required answer policy. At 262K, this pipeline reaches 82%/93% single-hop and 27%/41% multi-hop with gpt-4o-mini/gpt-4o, exceeding every result reported in the MAB v3 FactConsolidation comparison. This is a task-level result, not a claim that the evaluated memory architectures are broadly inferior. A controlled whole-pipeline comparison, with identical backbone, retrieved top-10 evidence, chunking, and n=100 per cell, improves single-hop accuracy by 10.8 percentage points (pp) on average and 21 pp at 262K. A targeted comparison using the same extraction setup shows that changing only the final policy executor contributes 2.0 pp on average and 0 pp at 262K. Most of the gain therefore comes from separating evidence identification from final policy execution rather than from the freshness operator itself. A LongMemEval check finds no significant overall advantage (26/45 versus 29/45; paired exact McNemar p=0.45), bounding the result to current-value questions with explicit version metadata. The evidence identifies post-retrieval assembly as a distinct reliability boundary between retrieval and answer generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。