通过稀疏内存提升线性RNN的上下文记忆能力，性能接近Transformer且更高效。
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
- 用稀疏读写机制扩展线性RNN的隐藏状态容量，保持计算量不变。
- 相同参数和计算量下，长文本召回和上下文学习性能显著提升。
- 可学习初始状态，适用于常识推理与复杂任务，适合大模型设计参考。
线性注意力模型具有固定的隐藏状态大小和每标记固定计算量，但受限于状态规模，在长上下文记忆方面落后于基于softmax注意力的Transformer架构。增加线性注意力的状态大小虽能提升召回性能，但会带来更高的浮点运算量（FLOPs）。本文提出稀疏增量记忆（Sparse Delta Memory, SDM），通过稀疏寻址方案将门控线性RNN的隐藏状态容量提升至原有数量级之上。SDM在门控增量网络（Gated DeltaNet）基础上，以稀疏读写方式替代密集的键值外积，实现对大型显式内存的访问。实验表明，在同等FLOPs和参数量约束下，更高容量的状态记忆显著改善了上下文学习与长上下文检索任务的表现。此外，通过学习SDM内存的初始状态并将其作为参数化记忆，模型在多种通用知识与推理任务上进一步取得提升。
原文摘要 · Abstract (English)
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。