混合两种记忆机制，让模型既快又准地处理长序列。
Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
- 用软注意力与动态权重结合，融合两种记忆方式。
- 在语言建模和强化学习中均优于单一记忆系统。
- 适合需要长程依赖与精准记忆的任务场景。
我们为通用序列处理神经网络设计了混合记忆架构，将基于软注意力的键值记忆（KV-memory）与通过动态突触调制实现的快速权重记忆（FW-memory）相结合——前者是二次型变换器的核心，后者是线性变换器的核心。这两种记忆系统具有互补但各自受限的特性：KV-memory 能精确检索，但序列长度呈二次复杂度增长；而 FW-memory 可支持任意长序列并具备更强表达能力，但牺牲了精确回忆。我们提出了三种融合方法，区别在于输入信息如何及何时传递给各子系统，以充分发挥两者优势。我们在语言建模、检索任务上训练了 340M 和 1.3B 参数模型，并在合成算法任务中验证了不同混合策略的优劣，还评估了其在部分可观测环境下的强化学习表现。结果表明，经过精心设计的混合系统可克服单一组件的局限，为神经记忆系统的设计提供了新思路。
原文摘要 · Abstract (English)
We develop hybrid memory architectures for general-purpose sequence processing neural networks, that combine key-value memory using softmax attention (KV-memory) with fast weight memory through dynamic synaptic modulation (FW-memory) -- the core principles of quadratic and linear transformers, respectively. These two memory systems have complementary but individually limited properties: KV-memory offers precise retrieval but is constrained by quadratic complexity in sequence length, while FW-memory supports arbitrarily long sequences and enables more expressive computation but sacrifices precise recall. We propose and compare three methods to blend these two systems into a single memory system, differing in how and when input information is delivered to each system, to leverage the strengths of both. We conduct experiments on general language modeling and retrieval tasks by training 340M- and 1.3B-parameter models from scratch, as well as on synthetic algorithmic tasks designed to precisely illustrate the benefits of certain hybrid methods over others. We also evaluate our hybrid memory systems on reinforcement learning in partially observable environments. Overall, we demonstrate how a well-designed hybrid can overcome the limitations of its individual components, offering new insights into the design principle of neural memory systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。