让大模型长时任务更高效,通过索引记忆避免信息丢失。
Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
- 用索引+摘要的外部记忆库替代原生上下文,保留完整历史
- 在长程任务中成功率提升,工作上下文减少60%以上
- 适合需要长期推理的复杂智能体,如自动化决策系统
大型语言模型代理在长时任务中受限于有限的上下文窗口。随着轨迹增长,将工具输出和中间推理保留在上下文中变得不可行:工作上下文过长,最终超出上下文预算,即使远期证据仍存在也难以使用。现有方法通常通过截断或生成摘要缩短上下文,但这些方法本质上有损,因压缩或丢弃了过去证据本身。我们提出 Memex,一种基于索引的经验记忆机制,通过压缩上下文而不丢弃证据来解决此问题。Memex 维持一个紧凑的工作上下文,包含结构化的简要摘要和稳定的索引,同时将完整的交互记录存储在外部经验数据库中,以索引为键。代理可自主决定何时解引用索引并恢复当前子目标所需的精确历史证据。我们通过强化学习框架 MemexRL 优化读写行为,采用针对上下文预算下索引记忆使用的奖励塑造,使代理学会总结什么、归档什么、如何索引以及何时检索。这比仅依赖摘要的方法显著降低信息损失。我们进一步提供理论分析,证明在有限解引用次数下,Memex 循环能保持决策质量,并在历史增长时仍维持有效的上下文计算。实验表明,在挑战性长时任务上,经 MemexRL 训练的 Memex 代理在显著减小工作上下文的同时提升了任务成功率。
原文摘要 · Abstract (English)
Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。