arXiv:2509.25140cs.AIcs.CL2025-09中稿 · ICLR被引 191

让智能体通过记忆自我进化,持续提升任务能力。

ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory

  • 构建推理记忆库,提取成功与失败经验中的通用策略
  • 测试时检索记忆并反哺新学习,实现能力渐进提升
  • 结合计算资源扩展,加速生成多样经验以优化记忆

随着大语言模型智能体在持久现实角色中的广泛应用,它们不断面临连续任务流。然而,现有方法难以从累积交互历史中学习,导致宝贵经验丢失和重复错误。本文提出ReasoningBank,一种新型记忆框架,可从智能体自评的成功与失败经历中提炼可泛化的推理策略。测试时,智能体从记忆库中检索相关记忆以指导交互,并将新学习成果回写,实现能力持续增强。在此基础上,我们引入记忆感知的测试时扩展(MaTTS),通过增加每项任务的计算资源,生成丰富多样的交互经验,提供更强对比信号以合成高质量记忆。更优的记忆反过来引导更高效的扩展,形成记忆与测试时扩展之间的强大协同。在网页浏览与软件工程基准上,ReasoningBank始终优于仅存储原始轨迹或成功路径的现有记忆机制,在效果与效率上均有提升;MaTTS进一步放大这些收益。结果表明,基于记忆的经验扩展是一种新的可扩展维度,使智能体能够自然涌现出自我演化行为。代码见:https://github.com/google-research/reasoning-bank。

原文摘要 · Abstract (English)

With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time. Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience. By allocating more compute to each task, the agent generates abundant, diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. The better memory in turn guides more effective scaling, establishing a powerful synergy between memory and test-time scaling. Across web browsing and software engineering benchmarks, ReasoningBank consistently outperforms existing memory mechanisms that store raw trajectories or only successful task routines, improving both effectiveness and efficiency; MaTTS further amplifies these gains. These findings establish memory-driven experience scaling as a new scaling dimension, enabling agents to self-evolve with emergent behaviors naturally arise. Our code can be found at https://github.com/google-research/reasoning-bank.

智能体记忆机制自我进化测试时扩展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。