通过可逆压缩实现记忆保留与检索的统一，提升大模型长程记忆能力。
R$^3$Mem: Bridging Memory Retention and Retrieval via Reversible Compression
- 用虚拟记忆标记压缩无限长历史，分层级提炼文档到实体信息
- 可逆架构支持从压缩数据还原原始内容，提升检索可靠性
- 轻量参数微调适配任意Transformer模型，适合对话等长程任务
记忆在真实场景中增强大语言模型性能至关重要。现有方案存在权衡：显式记忆需外部存储管理复杂且占用空间，隐式记忆依赖参数存储则难保证可靠检索。本文提出R$^3$Mem，一种通过可逆上下文压缩优化信息保留与检索的内存网络。该方法利用虚拟记忆标记压缩并编码无限长度的历史，结合分层压缩策略，从文档级到实体级逐级细化信息以提升多粒度融合效果。检索时采用可逆架构，通过反向调用模型从压缩信息重构原始数据。基于参数高效微调实现，可无缝集成至任意Transformer模型。实验表明，该设计在长上下文语言建模和检索增强生成任务中达到当前最优表现；在对话代理等长时交互任务中显著优于传统记忆模块，展现出下一代检索系统潜力。
原文摘要 · Abstract (English)
Memory plays a key role in enhancing LLMs' performance when deployed to real-world applications. Existing solutions face trade-offs: explicit memory designs based on external storage require complex management and incur storage overhead, while implicit memory designs that store information via parameters struggle with reliable retrieval. In this paper, we propose R$^3$Mem, a memory network that optimizes both information Retention and Retrieval through Reversible context compression. Specifically, R$^3$Mem employs virtual memory tokens to compress and encode infinitely long histories, further enhanced by a hierarchical compression strategy that refines information from document- to entity-level for improved assimilation across granularities. For retrieval, R$^3$Mem employs a reversible architecture, reconstructing raw data by invoking the model backward with compressed information. Implemented via parameter-efficient fine-tuning, it can integrate seamlessly with any Transformer-based model. Experiments demonstrate that our memory design achieves state-of-the-art performance in long-context language modeling and retrieval-augmented generation tasks. It also significantly outperforms conventional memory modules in long-horizon interaction tasks like conversational agents, showcasing its potential for next-generation retrieval systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。