让大模型在测试时通过记忆优化自我提升,不依赖人工规则。
From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

- 用强化学习训练记忆更新策略,端到端优化多轮决策
- 在德州扑克和剪刀石头布中分别获得1762和1590的最高等级分
- 无需修改主模型,可直接插入现有系统使用
大型语言模型代理在长时间运行场景中日益重要,测试时通过经验改进性能愈发关键。现有方法通常依赖人工设计的提示规则更新显式记忆,难以在多步决策中与下游目标保持一致。我们提出MemoPilot,一个可插拔的记忆协作者,通过强化学习显式训练记忆更新过程,以提升冻结版大模型在连续交互中的表现。我们将记忆更新建模为多轮决策问题,并采用多轮GRPO进行端到端优化。训练方案引入(i)每轮奖励信号和(ii)跨回放的上下文无关轮次级优势估计,实现更精细的信用分配和更稳定的多轮训练。我们在两个测试平台——多轮剪刀石头布(RPS)和限注德州扑克(LHE)上评估了MemoPilot。在两个环境中,它均显著优于强基线,在两者中均取得最高Elo评分(LHE: 1762,RPS: 1590),超越所有基线记忆方法及专有模型(包括DeepSeek-V3.2)。
原文摘要 · Abstract (English)
Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。