让大模型通过可训练图记忆学会总结经验,提升决策能力。
From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
- 用多层图结构抽象智能体行为路径,形成可优化的策略记忆。
- 通过强化学习动态调整记忆权重,显著提升任务推理表现。
- 适合需要长期规划与策略学习的自主智能体研究者使用。
基于大语言模型(LLM)的智能体在复杂开放环境中展现出强大的自主任务求解能力。提升其推理能力的关键在于更好地利用过往经验指导当前决策。然而,现有方法要么依赖训练中隐式记忆(存在灾难性遗忘和解释性差的问题),要么依赖提示中的显式记忆(缺乏适应性)。本文提出一种面向智能体的可训练多层图记忆框架,将原始智能体轨迹抽象为状态机中的结构化决策路径,并进一步提炼为高层、人类可理解的战略元认知。为实现记忆自适应,我们设计了一种基于强化学习的权重优化机制,根据下游任务的奖励反馈估计每条元认知的实际效用。优化后的策略通过元认知提示动态融入LLM智能体的训练循环。实验表明,可学习的图记忆能实现稳健泛化,显著提升智能体的战略推理性能,并在强化学习训练中持续带来收益。
原文摘要 · Abstract (English)
Large Language Models (LLMs) based agents have demonstrated remarkable potential in autonomous task-solving across complex, open-ended environments. A promising approach for improving the reasoning capabilities of LLM agents is to better utilize prior experiences in guiding current decisions. However, LLMs acquire experience either through implicit memory via training, which suffers from catastrophic forgetting and limited interpretability, or explicit memory via prompting, which lacks adaptability. In this paper, we introduce a novel agent-centric, trainable, multi-layered graph memory framework and evaluate how context memory enhances the ability of LLMs to utilize parametric information. The graph abstracts raw agent trajectories into structured decision paths in a state machine and further distills them into high-level, human-interpretable strategic meta-cognition. In order to make memory adaptable, we propose a reinforcement-based weight optimization procedure that estimates the empirical utility of each meta-cognition based on reward feedback from downstream tasks. These optimized strategies are then dynamically integrated into the LLM agent's training loop through meta-cognitive prompting. Empirically, the learnable graph memory delivers robust generalization, improves LLM agents' strategic reasoning performance, and provides consistent benefits during Reinforcement Learning (RL) training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。