MELODI通过分层压缩实现长文本高效处理，内存减少8倍仍保持高性能。
MELODI: Exploring Memory Compression for Long Contexts
- 分层压缩短时与长时记忆，跨窗口平滑衔接
- 在64K长记忆基线下性能更优，内存降低8倍
- 适合需要长上下文但资源受限的场景
我们提出MELODI，一种新型记忆架构，用于在短上下文窗口下高效处理长文档。其核心思想是将短期和长期记忆以分层压缩方式在不同网络层和上下文窗口间实现。具体而言，短期记忆通过多层间上下文窗口的递归压缩实现，确保窗口间的平滑过渡；而长期记忆则在单个中间层内进一步压缩，并聚合各窗口信息，有效整合整个历史的关键内容。相较于强基线模型——采用密集注意力机制的大规模长期记忆（64K键值对）的Memorizing Transformer，MELODI在多个长上下文数据集上表现更优，同时内存占用减少8倍。
原文摘要 · Abstract (English)
We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。