用稀疏记忆实现长文本建模，速度更快且更省资源。
Parallel Causal Associative Fields: Gated Sparse Memory for Long-Context Language Modeling
- 构建并行因果关联场，通过哈希存储和检索关键上下文
- 在2048长度下，困惑度比标准Transformer降低11.18和1.39
- 适合需要高效处理长文本的场景，如文档理解与生成
Transformers 通过直接的词元间通信路径实现强大语言建模能力，但因果自注意力随上下文长度呈平方增长。循环和状态空间模型虽降低了计算成本，却将历史压缩为顺序更新的固定大小状态。本文提出第三种范式：基于因果后继记录的并行内容寻址记忆。提出的平行因果关联场（PCAF）将上下文窗口内的局部记录写入哈希桶，检索当前查询的有限候选集，形成对后继词元的稀疏缓存分布，并通过可学习门控将其与参数化局部语言模型融合。该模型在保持稀疏长上下文访问的同时，避免了单一固定递归状态瓶颈。我们在 WikiText-103 和 PG-19 上进行全自回归预训练，使用分布式 Google Cloud TPU v4-32 pod。在 303M 参数、上下文长度 T = 2048 时，PCAF-semantic 在 WikiText-103 上达到 36.31 的困惑度，在 PG-19 上为 52.45，优于匹配的密集 Transformer（47.49、53.84）。PCAF-semantic 在 TPU pod 上同时处理 0.61–0.62M tokens/s，高于密集和局部注意力基线的 0.43M tokens/s。支持 41M 参数多种子扫描和单 GPU 组件消融实验表明，关联缓存、检索容量和可学习门控显著影响速度与质量权衡。
原文摘要 · Abstract (English)
Transformers achieve strong language modeling performance by providing direct token-to-token communication paths, but causal self-attention scales quadratically with context length. Recurrent and state-space models reduce this cost, yet compress history into sequentially updated fixed-size states. This paper studies a third primitive: a parallel content-addressed memory over causal successor records. The proposed Parallel Causal Associative Field (PCAF) writes local records from a context window into hash buckets, retrieves a bounded candidate set for the current query, forms a sparse cache distribution over successor tokens, and mixes that cache with a parametric local language model through a learned gate. The resulting model maintains sparse long-context access while avoiding a single fixed recurrent state bottleneck. We evaluate PCAF under full autoregressive pretraining on WikiText-103 and PG-19 using a distributed Google Cloud TPU v4-32 pod. At 303M parameters and context length T = 2048, PCAF-semantic reaches 36.31 perplexity on WikiText-103 and 52.45 perplexity on PG-19, compared with 47.49 and 53.84 for a matched dense Transformer. PCAF-semantic simultaneously processes 0.61-0.62M tokens/s across the TPU pod, versus 0.43M tokens/s for dense and local attention baselines. Supporting 41M-parameter multi-seed sweeps and single-GPU component ablations show that the associative cache, retrieval capacity, and learned gate materially affect the speed-quality trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。