单层模型也能高效语言建模，突破深度依赖传统。
A Single-Layer Model Can Do Language Modeling
- 用单个循环模块和共享内存实现语言建模，摒弃多层堆叠。
- 130M参数单层模型在FineWeb-Edu上达18.06困惑度，接近深层模型。
- 可直接观测状态向量结构，揭示记忆分快慢池的自发形成机制。
现代语言模型通过堆叠多层来扩展深度，每层拥有独立状态（如Transformer的键值缓存、Mamba、GDN、RWKV、xLSTM中的矩阵）。生物系统则更依赖递归而非堆叠。我们探讨语言建模中这种结构能走多远。提出接地预测网络（GPN）：通过单一循环块，在每一步反复访问同一个状态向量——一个前馈网络与一个共享矩阵记忆。在130M参数下，单层GPN+M在FineWeb-Edu上的困惑度为18.06，较12层Transformer++（16.05）仅差13%，较10层GDN（15.34）差18%；双层版本差距缩小至6%/11%。虽未超越深层基线，但因工作上下文仅为单一向量，可直接分析其几何结构：存在持续的默认词方向、数十词内容感知范围，以及自发分裂为快速与慢速存储池的记忆头。
原文摘要 · Abstract (English)
Modern language models scale depth by stacking layers, each holding its own state - a per-layer KV cache in transformers, a per-layer matrix in Mamba, Gated DeltaNet (GDN), RWKV, and xLSTM. Biological systems lean heavily on recurrence rather than on stacking. We ask how far that shape can go on language modeling. We propose Grounded Prediction Networks (GPN): one state vector revisited at every step through a single recurrent block - one FFN, one shared matrix memory. At 130M parameters, a 1-layer GPN+M reaches FineWeb-Edu perplexity 18.06, within 13% of a 12-layer Transformer++ (16.05) and 18% of a 10-layer GDN (15.34); a 2-layer variant closes the gap to 6%/11%. We do not match the deep baselines. Because the working context is a single vector, we can directly inspect its geometry: a persistent default-token direction, a content-bearing horizon of tens of tokens, and memory heads that split spontaneously into fast and slow retention pools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。