无需注意力机制的模型也能高效生成文本,且小数据下表现更优。
Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

- 用无注意力架构和多尺度延迟状态实现自回归文本生成
- 32MB数据即达512MBTransformer效果,50万参数时比特率低至1.84
- 生成质量关键在解码策略,非模型结构,适合轻量级部署
Kathleen系列论文1-2表明,仅用字节级、无注意力架构(基于波表编码器与多尺度延迟状态)在约45万至70万参数下即可匹配强基线分类性能,且无需预训练。本文探讨该架构是否可用于生成。首先,语言建模实验(WikiText-103,原始UTF-8,无分词器)显示:在2-512MB各数据规模下,延迟模型均优于参数相当的Transformer,例如512MB时比特率1.84比2.04更低;而Transformer需超过512MB才能达到该模型在32MB下的表现。其次,提出FORM DISTANCE,一种非参数化、抗游戏化的“文本像不像”评估工具:九维人类文本统计特征构成参考云,五种构造伪文本均被拒绝。第三,解码策略决定生成质量——扩大采样器使模型距离从3.17降至1.52,引入检索增强解码(不需训练)进一步降至1.14,增益源于稀疏短语本身而非选择门;但该增益有严格边界:短语必须来自模型自身训练语料,外源40倍大语料无帮助,这一现象也见于注意力模型,符合上下文整合是规模能力的体现。此外,四项架构改进无效,计算词典仅用五分之一参数即达学习词表94%的Top-1准确率。所有实验离线运行,可复现于免费Kaggle T4环境。
原文摘要 · Abstract (English)
Papers 1-2 of the Kathleen series showed that a byte-level, attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining. We ask whether the same ingredients can generate. (1) Scaling: on byte-level language modeling (WikiText-103, raw UTF-8, no tokenizer), the reverberant model beats a parameter-matched transformer at every dataset scale measured (2-512 MB), e.g. 1.84 vs 2.04 bits/byte at 512 MB with ~0.5M parameters; the transformer needs more than 512 MB to match what the attention-free model learns from 32 MB. (2) Measurement: we introduce FORM DISTANCE, a non-parametric, gaming-resistant instrument for "reads like text": nine statistical axes of human text define a reference cloud, and five constructed fakes are all rejected. (3) Generation: decoding policy dominates architecture -- widening the sampler halves the same model's distance (3.17 to 1.52), and a retrieval-augmented decoding scheme takes the frozen model further (1.52 to 1.14) with no training step involved; the ablation attributes the gain to the sparse phrase dose itself, not the selection gate. The gain has a sharp boundary condition: the phrases must come from the model's own training corpus -- a 40x larger foreign library helps not at all, an effect the attention twin shares, consistent with in-context integration being a capability of scale. We also report four architectural additions that did not help, and a computed lexicon reaching 94% of a learned table's top-1 accuracy at one fifth of the parameters. Everything runs offline; all experiments are reproducible on a free Kaggle T4.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。