arXiv:2607.28272cs.AI2026-07被引 1

让大模型像人一样重构记忆,而非机械复述,提升决策适应性。

MemHarness: Memory Is Reconstructed, Not Replayed

论文配图:MemHarness: Memory Is Reconstructed, Not Replayed
图 1 · 摘自论文原文
  • 基于当前情境动态重构过往经验,而非直接复用
  • 在ALFWorld和WebShop上显著优于传统强化学习与静态记忆方法
  • 特别适合处理分布外场景,增强模型内在推理能力

检索过往经验已成为提升大语言模型代理性能的常见策略。然而,现有记忆增强型代理大多将检索到的经验视为静态记录,不加区分地将其注入上下文,忽视了存储经验的抽象性与决策时刻具体情境之间的差异,常导致负向迁移。相比之下,人类很少原样回忆过去,而是会重组并调整记忆以契合当前情境。受此启发,我们提出MemHarness框架,使LLM代理能够根据当前状态主动提炼和重构过往经验。在每个决策步骤中,统一策略模型会基于当前状态对检索到的经验进行批判性重构,生成与上下文一致的指导信息后再执行动作。这种重构能力通过端到端的GRPO训练自然涌现。在ALFWorld和WebShop上的实验表明,MemHarness显著优于纯强化学习及静态记忆增强基线,在分布外(OOD)场景下表现出强鲁棒性。进一步分析显示,该重构目标不仅能防止负向迁移,还能在训练过程中充当隐式引导,从根本上提升代理的内在推理能力。

原文摘要 · Abstract (English)

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.

大模型代理记忆重构推理增强RLHF

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。