解决代码助手记忆中的过时信息问题,让工具能自动识别最新代码变更。
Temporal Validity on Real Software Histories: Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixes
- 提出基于时间有效性记忆的结构化存储机制,区分新旧代码状态。
- 在真实GitHub修复数据上,准确率从RAG的57%提升至91%。
- 特别适合需要精确历史状态追踪的代码生成与维护场景。
检索增强生成(RAG)缺乏时间模型:当代码在会话中发生变更(如函数重命名、接口迁移、依赖升级),旧值和新值相似度相近,RAG无法判断当前值,常返回过时信息。本文在真实软件历史中验证了一种确定性(主体、关系、对象)的更新记忆机制。从707个真实GitHub问题(SWE-bench Lite + Verified)中提取130个干净的原子状态转换(仅一个可识别值从修复前变为修复后),并去除标记信息(仅值不同)。实验表明,MemStrata达到0.91的答案准确率,远超RAG的0.57-0.59;且强制回答时,RAG仍以36-38%概率返回过时值(即使使用LLM重排序也无效),而MemStrata将其降至约0。推理延迟仅2.1秒,远低于重排序器的18秒。研究明确限定范围:仅约18%的真实修复属于此类干净原子转换;其余情况的提取覆盖是后续工作的独立课题。研究期间发现并修复了一个真实产品缺陷(大小写不敏感的值比较),其核心机制(对干净代码变更的确定性更新准确率)得到验证并保持。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has no model of time: when a fact changes across a coding session - a function is renamed, an endpoint moves, a dependency is bumped - RAG retrieves both the old and new value with near-identical similarity and cannot tell which is current, so it serves the superseded value. Paper 1 showed, on synthetic single-value benchmarks, that a deterministic (subject, relation, object) supersession memory eliminates this failure. Here we validate it end-to-end on real software history. From 707 real GitHub issues (SWE-bench Lite + Verified) we extract 130 clean atomic state transitions, a fix that changes one identifiable value from a pre-fix to a post-fix form, and render each marker-free (the stale and current statements differ only in the value). On this set, MemStrata reaches 0.91 answer accuracy versus RAG's 0.57-0.59; and, the structural result, when forced to answer RAG serves the superseded value 36-38% of the time (an LLM reranker does not help) while MemStrata drives this to ~0, at RAG retrieval latency (~2.1 s vs ~18 s for the reranker). We are explicit about scope: only ~18% of real fixes are clean atomic transitions; Paper 2 isolates the memory mechanism on that class, and extraction coverage of the remaining fixes is the orthogonal problem we defer to follow-on work. A real product bug surfaced and was fixed during the study (a case/punctuation-insensitive value comparison), with the moat property (deterministic-supersession accuracy on clean code mutations) preserved and verified.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。