研究发现,压缩对话记忆会丢失时间信息,导致时序问题解答变差。
The Sleeping Agent: What Gist-Based Context Compression Loses and Why
- 用类睡眠记忆整合机制分析对话压缩效果
- 压缩后时序题准确率大幅下降,事实与多跳推理题反而提升
- 修改提示词可精准恢复时间信息,适合时序任务研究者
基于要点的上下文压缩在长时语言模型代理中常见,但其对不同类型记忆检索的影响尚不明确。本文采用受生物启发的显著性加权整合(SWC)框架作为诊断工具,分析压缩在何时有益、何时有害。在10个LoCoMo对话的1,501个非对抗性文本问题上(温度为0),发现:压缩显著优于截断,适用于多跳推理和单跳事实问答;但时序问题在压缩下仍显著更难,压缩条件得分远低于全上下文参考。原因在于:压缩提示保留事件结构但丢弃日期时间。跨所有对话的分析显示,仅修改一句提示,时间表达保留率从3.05%提升至62.39%(约20倍),而命名实体与事件保留率变化极小(分别仅升1.02倍和1.11倍),证明该修复具有高精度。此修改使类别2(时序)问题的裁判准确率提升+0.314([0.254, 0.375])。代码与结果见:https://github.com/kyrkewood/sleeping-agent。
原文摘要 · Abstract (English)
Gist-based context compression---summarising older conversation history into compact representations---is a common approach in long-horizon language model agents, yet its effect on different types of memory retrieval is poorly understood. We use Salience-Weighted Consolidation (SWC), a biologically-inspired compression framework motivated by sleep-based memory consolidation, as a diagnostic probe to study when gist compression helps and when it hurts. SWC scores conversation history by salience, partitions it into priority tiers, and applies structured gist abstraction to mid-priority content. Evaluating four conditions on all ten LoCoMo conversations---1,935 matched text-only questions in total, 1,501 used in the primary aggregate after excluding Category 5 (adversarial) questions---at temperature 0, we find a consistent task-type interaction: gist compression substantially outperforms truncation on multi-hop reasoning and single-hop factual questions, but temporal questions remain substantially harder under compression, with compressed conditions scoring well below the full-context reference on the conversations where both are evaluated. We trace this failure to a specific mechanism: the gist abstraction prompt preserves relational and event structure while discarding dates and times. A preservation analysis across all ten conversations confirms the mechanism: an approximately 20-fold increase in temporal expression preservation (3.05% to 62.39%) with a one-sentence prompt modification, while named entity and event preservation rates barely change (x1.02 and x1.11), demonstrating that the fix is a precision instrument. The prompt modification recovers +0.314 [0.254, 0.375] judge accuracy on category-2 (temporal) questions in the matched set. Code and results: https://github.com/kyrkewood/sleeping-agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。