用分层局部注意力模拟语言模型,揭示计算机制对性能的影响。
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

- 设计分层文本神经元胞自动机,通过三级窗口逐步扩大注意力范围。
- 最优配置下困惑度达60.3,性能接近但未超越同参数量Transformer。
- 迭代次数和门控机制是关键,可调节推理时的计算步数。
我们定义了TextNCA,一种基于1D因果滑动窗口注意力的神经元胞自动机语言建模方法,并研究其分层变体。该模型在WikiText-103数据集上使用约3000万参数、6万次训练步骤,包含三级窗口大小分别为8、32、128的层级结构,每级执行共享权重的迭代步数$T_s$。尽管其困惑度(PPL)为60.3,低于参数匹配的6层和12层Transformer(分别为52.8和44.7),但本文将其作为分析工具而非替代方案。实验表明,性能主要受窄到宽的层级调度驱动:非迭代的滑动窗口Transformer若采用相同调度,仅比迭代模型差+4.1 PPL;而反向、打乱或破坏单调顺序则导致+16.7至+70.8 PPL的显著下降。迭代带来有限且有界的好处,在$T_s=4$时达到最优,超过后呈U形退化。GRU门控与每步学习嵌入对这一优势至关重要;随机训练$T_s$可实现推理时的迭代步数控制,但代价是更高绝对困惑度。本工作旨在系统评估神经元胞自动机式计算中哪些成分真正影响语言建模表现。
原文摘要 · Abstract (English)
Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define \textsc{TextNCA}, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows $w \in \{8, 32, 128\}$ and $T_s$ shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched Transformer at this scale (Hier-TextNCA $60.3$ vs.\ Transformer-6L $52.8$ and Transformer-12L $44.7$ PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window Transformer that reuses the same schedule comes within $+4.1$ PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between $+16.7$ and $+70.8$ PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at $T_s{=}4$ and a U-shaped degradation beyond it. The GRU gate and learned per-step embeddings are required for that benefit to appear, and training with random $T_s$ yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。