诊断长文本记忆系统中写入与检索的瓶颈,发现写入环节问题更严重。
WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems
- 设计四条件诊断法,分离评估写入与检索性能
- 六种基线中四款在写入阶段丢失信息更严重
- 提出EPC方法,提前预测问题并保留关键证据
长上下文记忆系统在固定资源预算下常失效,但端到端评估无法区分是压缩时丢弃了信息,还是保存了但未被检索。本文提出四条件诊断协议:在固定阅读器下测试截断完整上下文(TFC)、理想证据(OE)、完整存储记忆(CSM)和检索记忆(RM)。在固定预算的LongMemEval设置下,多数基线的写入侧差距大于检索侧,六种基线中有四种在默认诊断阈值下显著表现为写入主导型。受此启发,提出预期预测压缩(EPC),利用大模型预判未来可能的问题,在写入阶段即保留最小必要支持证据,同时保持检索阶段不变。在包含500个问题的LongMemEval测试中,使用三种阅读器(GPT-5.2、Claude Sonnet 4、Gemini 2.5 Pro),EPC在所有系统中取得最高CSM得分(0.49,优于最强基线摘要模型的0.44),将写入差距降至0.04,而检索差距与其他基于大模型的系统相当。结果表明,在该基准和评估设置下,提升写入阶段的保留内容是性能提升的关键路径。
原文摘要 · Abstract (English)
Long-context memory systems often fail under fixed budgets, but end-to-end evaluation does not reveal whether evidence was discarded during compression or preserved but never retrieved. We introduce a four-condition diagnostic protocol that evaluates a fixed reader under truncated full context (TFC), oracle evidence (OE), complete stored memory (CSM), and retrieved memory (RM). Under this fixed-budget LongMemEval setup, write-side gaps exceed retrieval-side gaps for most tested baselines, with four of six baselines robustly write-dominant under our default diagnosis margin. Motivated by this diagnosis, we propose Expected Predictive Compression (EPC), which moves the key decision--what information to retain--to write time by using an LLM to anticipate likely future questions and preserve the minimal supporting evidence under the token budget, while leaving retrieval unchanged at question time. Across all 500 LongMemEval questions with three readers (GPT-5.2, Claude Sonnet 4, Gemini 2.5 Pro), EPC achieves the highest CSM scores among all systems (0.49 vs. 0.44 for Summary (LLM), the strongest baseline), reducing Delta_write to 0.04 while leaving Delta_retr comparable to other LLM-based systems. These results suggest that, on this benchmark and evaluation setup, improving what the write stage preserves is a key avenue for performance gains in the tested systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。