分离状态存储与预测计算，提升Transformer语言模型效率
The State-Prediction Separation Hypothesis
- 设计双流Transformer，分别处理状态记忆和下一个词预测
- 在多个规模实验中，验证损失更低，下游任务平均提升2-3个百分点
- 适合追求高效训练与推理的NLP研究者与工程师
Transformer使用相同的前向计算流同时预测下一个词并存储对未来预测有用的状态。我们提出“状态-预测分离假设”：将这两种功能解耦可提升语言建模性能。为此设计了一种使用两条独立计算流的Transformer变体，并在多种规模下进行了预训练实验。结果表明，状态-预测分离在数据和计算效率上均表现更优，验证损失更低，在下游任务上平均比标准Transformer提升2–3个百分点。通过大量实证分析排除了潜在混淆因素，证实了该设计带来的梯度本质差异。
原文摘要 · Abstract (English)
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction separation hypothesis}: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2--3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。