研究模型升级时记忆能否保留,发现固定结构记忆最可靠,压缩记忆易失效。
Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

- 用四种方式存储记忆:原始文本、分块检索、自然语言笔记、固定模式知识图谱
- 固定模式知识图谱迁移后准确率变化仅0.0004±0.0020,压缩笔记则波动达±13.28个百分点
- 建议保留原始历史用于修复,避免仅靠压缩记忆尝试恢复
模型升级常见,但记忆迁移却极少被系统研究。我们对比了四种记忆存储方式在模型更换后的表现:长期上下文原始文本(LC-RAW)、分块检索增强生成(RAG)、模型压缩为自然语言笔记(NOTES)以及固定模式知识图谱(KG-fixed)。实验采用48个合成历史,随机答案代码,精确评分,使用两个参数少于100亿的开源模型。结果显示,固定模式结构迁移最稳定,准确率变化仅为+0.0004±0.0020;而压缩笔记存在强模型依赖,迁移方向不同导致准确率变化高达+9.91或-13.28个百分点。在RAG中,混合嵌入仅带来4.96点提升,远低于全重嵌入的11.90点增益。诊断分析表明,80%的笔记损失源于初始构建时信息丢失,81%的检索失败由召回问题导致。仅修复存储无法使笔记恢复至90%性能,而保留原始数据可在一种方向上实现34/48的成功恢复。研究强调需针对性测试迁移路径、隔离嵌入空间,并保留源历史以支持记忆修复。
原文摘要 · Abstract (English)
Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only $+0.0004 \pm 0.0020$ following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by $+9.91$ or $-13.28$ percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% ($0.467 \pm 0.014$) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% ($0.364 \pm 0.012$) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。