自改进大模型因记忆评分膨胀,反而反复犯错,新方法可纠正这一问题。
Memory Reward Inflation in Self-Improving LLM Agents

- 用外部记忆存储经验,以模型自评分数作为奖励信号。
- 自评分数会错误放大错误记忆,导致模型越自信越犯错。
- 提出LUCID算法,在SQL生成任务上提升准确率至56.9%。
自改进的大语言模型代理通过外部记忆存储经验,不更新参数,仅依据存储的评分调整行为。该评分作为隐式策略的奖励信号,但部署时缺乏真实标签,评分由模型自身给出,引发‘回声鸿沟’:错误记忆因高自信而被过度奖励,错误随记忆累积且无法识别。我们证明,误差独立性假设(EIA)是纠正膨胀的必要条件——信号必须既反映真实值,又与记忆偏差去相关。即使在仅按相似度检索的部署场景中,膨胀依然存在。提出的无答案去膨胀算法LUCID在BIRD文本到SQL基准上实现端到端提升,执行准确率达56.9%,高于自评代理(54.0%)和无记忆基线(52.4%),平均增益+2.9点。
原文摘要 · Abstract (English)
Self-improving LLM agents increasingly learn from experience without updating any weights. Each episode is stored in an external memory, scored, and retrieved for similar future tasks to shape later behavior. Viewed through a reward lens, the stored score is a proxy reward for an implicit, non-parametric policy. Each retrieved episode then becomes a policy-improvement step whose reliability hinges on how that score is produced. In deployment, ground-truth labels are unavailable, so the stored reward is at best an LLM assessment. This substitution creates a failure mode, the *Echo Gap*, across the memory-based self-improving agents and model families studied. Incorrect episodes receive inflated rewards; thus, the agent preferentially reuses the very mistakes it has most confident in. Because the error compounds through memory rather than averaging out and the confirming judge's errors remain correlated with the original self-grading bias, so it cannot identify which memories are overvalued. The missing property is formalized as the *Error-Independence Assumption* (EIA), which we prove is a *necessary* condition for correcting the inflation, not merely a description of a good verifier: a usable signal must track truth *and* decorrelate its error from the memory bias, and the recoverable payoff is a closed-form function of exactly those two quantities. We further show the inflation compounds not only when retrieval ranks by the stored score but also under plain similarity retrieval which is the regime the deployed agent uses. Finally, the answer-free de-inflation algorithm LUCID delivers a consistent end-to-end gain on the BIRD text-to-SQL benchmark. It raises execution accuracy to $56.9\%$, above both a Memento-style self-graded agent ($54.0\%$, a $+2.9$-point mean gain across seeds) and a memory-less agent of identical architecture ($52.4\%$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。