arXiv:2506.15841cs.CLcs.AI2025-06被引 195

让智能体用固定内存完成长时间多轮任务,效率与性能双提升

MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents

论文配图:MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
图 1 · 摘自论文原文
  • 通过强化学习构建紧凑内部状态,融合记忆与推理
  • 在多跳问答任务中性能提升3.5倍,内存减少3.7倍
  • 适合长时交互、资源受限的智能体应用

现代语言代理需处理长时序、多轮交互任务,包括外部信息检索、环境观测适应和相互依赖的问题回答。然而,多数大模型系统采用全上下文提示,不断累积历史对话,导致内存无限制增长、计算成本上升,并在长输入下性能下降。我们提出MEM1,一种端到端强化学习框架,使代理在长多轮任务中保持恒定内存。每轮中,MEM1更新一个紧凑的共享内部状态,联合支持记忆整合与推理。该状态融合先前记忆与新环境观测,同时策略性丢弃无关或冗余信息。为支持更真实、组合式的训练环境,我们提出一种简单有效的方案,通过组合现有数据集生成任意复杂任务序列。在三个领域(内部检索问答、开放域网页问答、多轮网购)的实验表明,在16个目标的多跳问答任务中,MEM1-7B相比Qwen2.5-14B-Instruct性能提升3.5倍,内存使用降低3.7倍,且可泛化至训练跨度之外。结果证明,以推理驱动的记忆整合是训练长时交互代理的可扩展替代方案,实现效率与性能的协同优化。

原文摘要 · Abstract (English)

Modern language agents must operate over long-horizon, multi-turn interactions, where they retrieve external information, adapt to observations, and answer interdependent queries. Yet, most LLM systems rely on full-context prompting, appending all past turns regardless of their relevance. This leads to unbounded memory growth, increased computational costs, and degraded reasoning performance on out-of-distribution input lengths. We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with constant memory across long multi-turn tasks. At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning. This state integrates prior memory with new observations from the environment while strategically discarding irrelevant or redundant information. To support training in more realistic and compositional settings, we propose a simple yet effective and scalable approach to constructing multi-turn environments by composing existing datasets into arbitrarily complex task sequences. Experiments across three domains, including internal retrieval QA, open-domain web QA, and multi-turn web shopping, show that MEM1-7B improves performance by 3.5x while reducing memory usage by 3.7x compared to Qwen2.5-14B-Instruct on a 16-objective multi-hop QA task, and generalizes beyond the training horizon. Our results demonstrate the promise of reasoning-driven memory consolidation as a scalable alternative to existing solutions for training long-horizon interactive agents, where both efficiency and performance are optimized.

长时交互记忆优化强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。