提出可压缩可扩展的循环记忆架构，解决长序列建模中的内存与效率难题。
Towards Compressive and Scalable Recurrent Memory
- 基于HiPPO框架将历史序列视为连续信号，实现在线最优压缩编码
- 在32k以上长序列任务中性能超越基线，内存占用低16倍且速度更快
- 设计解耦结构支持测试时注入先验知识，适合需要灵活适应的场景
Transformer在处理长上下文时面临注意力机制的二次方计算瓶颈。现有引入循环记忆的方法在理论原则与实际可扩展性之间存在根本权衡。为此，本文提出Elastic Memory，一种基于HiPPO框架的在线函数逼近记忆架构。该方法将历史序列视为连续信号样本，通过最优在线压缩将其编码为固定大小的记忆状态。检索时采用灵活的多项式采样机制，从压缩状态重建历史摘要。Elastic Memory在三个领域的32k+长序列数据集上持续优于基线。在参数量相同情况下，其内存使用仅为Memorizing Transformer的1/16，且性能更优；相比Melodi，在所有记忆尺寸下均胜出，即使后者多出30%参数。模型规模扩大至4倍时，Elastic Memory仍保持领先，且显著快于Melodi。此外，其解耦设计支持测试时注入归纳偏置以提升性能。
原文摘要 · Abstract (English)
Transformers face a quadratic bottleneck in attention when scaling to long contexts. Recent approaches introduce recurrent memory to extend context beyond the current window, yet these often face a fundamental trade-off between theoretical principles and practical scalability. To address this, we introduce Elastic Memory, a novel memory architecture grounded in the HiPPO framework for online function approximation. Elastic Memory treats historical sequence as samples from continuous signals, applying optimal online compression to encode them into a fixed-size memory state. For retrieval, we propose a flexible \textit{polynomial sampling} mechanism that reconstructs a history summary from this compressed state. Elastic Memory consistently outperformed baselines on long-context (32k+) datasets across three domains. With equal parameters, it beat Memorizing Transformer by 16x memory and outperformed Melodi at all memory sizes, even when Melodi had 30% more parameters. When scaling model size, Elastic Memory stayed ahead of all baselines and was significantly faster than Melodi at 4x size. Furthermore, its decoupled design allows for injecting inductive biases at test-time to boost performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。