让大模型长时记忆更高效:图文混合组织,节省上下文空间
MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents

- 用文本、图像或双通道方式组织记忆块,按需分配
- 在四个基准上减少记忆令牌数,提升任务表现
- 适合需要长期记忆的智能体系统,如复杂推理与交互
长时运行的大语言模型智能体依赖外部记忆存储和复用超出单个上下文窗口的信息,但持续积累的交互轨迹与有限的上下文容量之间存在根本矛盾。核心挑战不仅是检索相关记录,还包括在给定预算下选择必要证据并以合适模态组织。现有读取方法多采用文本或视觉形式:文本保留高保真度,但线性标记表示使不同重要性的内容以近乎均等成本争夺上下文;视觉读取将文本转为类文档图像,利用二维布局展现结构并突出重点,但渲染与压缩过程可能损失细粒度信息。为此,我们提出MEMO——一种面向长时程大模型智能体的多模态证据记忆组织方法。MEMO首先通过训练好的证据提取器筛选相关记忆块,形成含来源与呈现要求的证据单元;再由查询条件驱动的记忆管理器将每个单元分配至文本、视觉或双通道载体,并选择匹配证据结构的版式;最后确定性记忆构建模块生成文本包与视觉页面。记忆管理器通过离线阅读器反馈进行训练,使保留与呈现决策与下游使用目标对齐。我们在HotpotQA、2WikiMultiHopQA、LoCoMo和ALFWorld四个基准上评估,使用多种阅读器后端。结果表明,MEMO以更少记忆令牌更高效地呈现记忆,提升下游任务性能,并在受限预算下构建更有效的工作记忆。
原文摘要 · Abstract (English)
Long-running LLM agents rely on external memory to store and reuse information beyond a single context window, yet there is a fundamental tension between the continuous accumulation of interaction trajectories and the limited context capacity. The key challenge in agent memory is therefore not only to retrieve relevant records, but also to select necessary evidence under a given budget and organize it in an appropriate modality. Existing memory readout methods mainly use textual or visual forms. Text preserves high fidelity, but its linear token representation makes contents with different importance compete for the limited context at nearly uniform unit cost. Visual readout renders text into document-like images, which can use two-dimensional layouts to expose structure and emphasize key information, but it may lose fine-grained details during rendering and compression. To address this issue, we propose MEMO, a multimodal evidence memory organization method for LLM agents. MEMO first uses a trained evidence extractor to select relevant memory blocks and form evidence units with source information and presentation requirements. A trained query-conditioned memory manager assigns each unit to a textual, visual, or dual-channel carrier and selects a layout that matches the evidence structure. A deterministic memory construction module then generates the textual package and visual pages. The memory manager is trained with feedback from an offline reader that measures the utility of the guided memory plan, so that retention and presentation decisions align with downstream usage. We evaluate MEMO on four benchmarks, HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld, with multiple reader backends. The results show that MEMO presents memory more efficiently with fewer memory tokens, improves downstream task performance, and builds more effective working memory under constrained budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。