arXiv:2601.23014cs.LGcs.CL2026-01被引 10

让记忆代理在长序列中高效学习,通过密集奖励提升训练效果。

Mem-T: Densifying Rewards for Long-Horizon Memory Agents

  • 用树状结构引导强化学习,将稀疏奖励转化为每步的密集监督。
  • 相比A-Mem和Mem0,性能最高提升14.92%,推理耗token减少约24.45%。
  • 适合需要长期记忆与高效推理的自主智能体系统研究者。

记忆代理通过内生方式自主管理记忆的处理、存储与检索,展现出更强的自主性与适应性。然而现有训练范式受限:代理需经历长序列记忆操作后才获得稀疏且延迟的奖励,难以实现记忆管理策略的端到端优化。为此,我们提出Mem-T,一个对接轻量级分层记忆数据库的自主记忆代理,可对流式输入动态更新并支持多轮检索。为有效训练长时记忆管理能力,进一步提出MoT-GRPO,一种树引导的强化学习框架,通过记忆操作树反向传播与事后信用分配,将稀疏终端反馈转化为密集的逐步监督,从而联合优化记忆构建与检索。大量实验表明,Mem-T(1)性能优异,相较A-Mem和Mem0最高提升14.92%;(2)经济高效,在有利的准确率-效率权衡前沿运行,相比GAM推理每查询减少约24.45%的token消耗,且不牺牲性能。

原文摘要 · Abstract (English)

Memory agents, which depart from predefined memory-processing pipelines by endogenously managing the processing, storage, and retrieval of memories, have garnered increasing attention for their autonomy and adaptability. However, existing training paradigms remain constrained: agents often traverse long-horizon sequences of memory operations before receiving sparse and delayed rewards, which hinders truly end-to-end optimization of memory management policies. To address this limitation, we introduce Mem-T, an autonomous memory agent that interfaces with a lightweight hierarchical memory database to perform dynamic updates and multi-turn retrieval over streaming inputs. To effectively train long-horizon memory management capabilities, we further propose MoT-GRPO, a tree-guided reinforcement learning framework that transforms sparse terminal feedback into dense, step-wise supervision via memory operation tree backpropagation and hindsight credit assignment, thereby enabling the joint optimization of memory construction and retrieval. Extensive experiments demonstrate that Mem-T is (1) high-performing, surpassing frameworks such as A-Mem and Mem0 by up to $14.92\%$, and (2) economical, operating on a favorable accuracy-efficiency Pareto frontier and reducing inference tokens per query by $\sim24.45\%$ relative to GAM without sacrificing performance.

记忆代理强化学习长时序高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。