arXiv:2602.11958cs.LGcs.CL2026-02

用稀疏地址实现海量记忆选择性访问,提升长序列建模效率与精度

RAM-Net: Expressive Linear Attention with Selectively Addressable Memory

  • 通过高维稀疏向量作为显式地址,实现对大规模内存的选择性读写
  • 状态规模可指数级扩展且不增加参数,有效减少信号干扰并提升检索准确率
  • 适合需要长距离依赖建模的场景,如文档理解、对话系统等

线性注意力架构虽具备高效推理能力,但将无限历史压缩为固定大小的记忆会限制表达能力并导致信息丢失。为此,我们提出随机访问记忆网络(RAM-Net),旨在弥合全注意力的表征能力与线性模型的记忆效率之间的差距。其核心思想是将输入映射为高维稀疏向量作为显式地址,使模型能够选择性地访问海量记忆状态。该设计实现了状态规模的指数级扩展而无需增加参数,显著缓解了信号干扰并提升了检索保真度。此外,固有的稀疏性确保了极高的计算效率,状态更新仅作用于极少数条目。大量实验表明,RAM-Net在细粒度长程检索任务中持续优于现有最优基线,在标准语言建模和零样本常识推理基准上也表现出竞争力,验证了其以显著降低计算开销捕捉复杂依赖关系的优越能力。

原文摘要 · Abstract (English)

While linear attention architectures offer efficient inference, compressing unbounded history into a fixed-size memory inherently limits expressivity and causes information loss. To address this limitation, we introduce Random Access Memory Network (RAM-Net), a novel architecture designed to bridge the gap between the representational capacity of full attention and the memory efficiency of linear models. The core of RAM-Net maps inputs to high-dimensional sparse vectors serving as explicit addresses, allowing the model to selectively access a massive memory state. This design enables exponential state size scaling without additional parameters, which significantly mitigates signal interference and enhances retrieval fidelity. Moreover, the inherent sparsity ensures exceptional computational efficiency, as state updates are confined to minimal entries. Extensive experiments demonstrate that RAM-Net consistently surpasses state-of-the-art baselines in fine-grained long-range retrieval tasks and achieves competitive performance in standard language modeling and zero-shot commonsense reasoning benchmarks, validating its superior capability to capture complex dependencies with significantly reduced computational overhead.

注意力机制长序列建模稀疏记忆高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。