让大模型通过记忆自我优化决策，提升任务适应性。
Memory-Driven Self-Improvement for Decision Making with Large Language Models
- 用记忆存储过往经验与价值，反哺大模型决策
- 在真实任务中性能提升超40%，跨任务泛化超75%
- 适合需要持续学习的智能体系统开发
大语言模型（LLMs）因其广泛先验知识，在序列决策任务中表现出色。然而，面对数据有限的具体任务，其通用知识往往不足，难以高效适配。为此，我们提出一种记忆驱动的自我改进框架，将大模型的通用先验知识与紧凑的领域特定经验记忆相结合。记忆保存历史交互与对应Q值，捕捉决策相关知识，辅助准确估值并优化大模型先验。优化后的先验生成更高回报轨迹，进一步丰富记忆，形成记忆与模型先验相互强化的自进化机制。实验表明，该方法显著优于传统强化学习与基于大模型的基线，例如在同分布任务上性能提升超过40%，在ALFWorld未见任务上泛化性能提升超过75%。
原文摘要 · Abstract (English)
Large language models (LLMs) have emerged as effective action policies for sequential decision-making (SDM) tasks due to their extensive prior knowledge. However, this broad yet general knowledge is often insufficient for specific decision-making tasks with limited task-related data, making it challenging to efficiently adapt LLMs to specific SDM tasks. To address this challenge, we propose a memory-driven self-improvement framework that combines LLM general prior knowledge with a compact memory of domain-specific experiences. Memory retains past interactions and associated Q-values, thereby capturing decision-relevant knowledge that facilitates accurate value estimation and informs the LLM prior refinement. The refined LLM prior, in turn, generates higher-reward trajectories that further enrich memory, forming a natural self-improvement framework where memory and LLM prior mutually reinforce each other. Experiments show that our memory-driven approach significantly outperforms both traditional RL and LLM-based baselines, e.g., improving performance by over 40\% on in-distribution tasks and over 75\% when generalized to unseen tasks in ALFWorld.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。