用编码器做下一个词预测，突破传统解码器限制。
ENTP: Encoder-only Next Token Prediction
- 用编码器结构实现下一词预测，避开因果注意力约束。
- 在计数任务中，编码器模型表现远超解码器模型。
- 适合算力充足场景，对上下文学习等任务更有效。
传统下一词预测采用仅解码器的Transformer与因果注意力机制，以高效复用键值。但若不考虑计算成本，是否仍应如此？本文提出编码器仅下一词预测（ENTP），对比其与解码器Transformer在表达能力与复杂度上的差异，揭示了在无限算力下使用编码器的潜在优势。我们引入$ ext{Count3}$任务，理论与实验证明：虽然ENTP能轻松完成该任务，解码器Transformer则无法做到。此外，我们在加法、上下文学习和语言建模等典型任务上实证显示，基于下一词预测的ENTP性能更优。
原文摘要 · Abstract (English)
Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were not compute-limited, should we still use decoder-only Transformers? In this work, we introduce Encoder-only Next Token Prediction (ENTP). We explore the differences between ENTP and decoder-only Transformers in expressive power and complexity, highlighting potential advantages of ENTP in settings with unbounded compute. We introduce the $\operatorname{Count3}$ task and show, both theoretically and experimentally, that while ENTP can perform this task easily, a decoder-only Transformer cannot. Finally, we empirically demonstrate the superior performance of ENTP across representative tasks where next-token prediction based Transformers can be evaluated, including addition, in-context learning, and language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。