用强化学习让大模型自主管理记忆，提升长程推理能力。
Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
- 设计双代理框架，通过强化学习学会存、改、删、不动等记忆操作。
- 仅用152个训练样本即在3个基准上超越基线，支持3B-14B模型规模。
- 适合需要长期记忆与复杂推理的智能体任务，如对话系统、知识问答。
大型语言模型（LLMs）在众多自然语言处理任务中表现出色，但本质上是无状态的，受限于有限的上下文窗口，难以进行长时序推理。现有方法虽引入外部记忆库，但多为静态、启发式驱动，缺乏对存储、更新、检索内容的可学习决策机制。本文提出 Memory-R1，一个基于强化学习（RL）的框架，通过两个专用代理实现动态记忆管理：记忆管理器学习结构化操作（添加、更新、删除、无操作）；回答代理预选并推理相关记忆条目。两者均采用结果导向的强化学习（PPO 和 GRPO）微调，在仅152个训练问答对下，性能优于强基线，并在三个基准（LoCoMo、MSC、LongMemEval）和多个模型规模（3B-14B）上实现良好泛化。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless, constrained by limited context windows that hinder long-horizon reasoning. Recent efforts to address this limitation often augment LLMs with an external memory bank, yet most existing pipelines are static and heuristic-driven, lacking a learned mechanism for deciding what to store, update, or retrieve. We present Memory-R1, a reinforcement learning (RL) framework that equips LLMs with the ability to actively manage and utilize external memory through two specialized agents: a Memory Manager that learns structured operations, including ADD, UPDATE, DELETE, and NOOP; and an Answer Agent that pre-selects and reasons over relevant entries. Both agents are fine-tuned with outcome-driven RL (PPO and GRPO), enabling adaptive memory management with minimal supervision. With only 152 training QA pairs, Memory-R1 outperforms strong baselines and generalizes across diverse question types, three benchmarks (LoCoMo, MSC, LongMemEval), and multiple model scales (3B-14B).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。