训练预测下一个词能让模型学会长距离结构，有理论证明。
Provable Long-Range Benefits of Next-Token Prediction
- 用RNN优化下一个词预测，可逼近真实文本分布。
- 生成的文本与真实文档在任意k个词内无法区分，且模型规模为k的多项式。
- 解释了为何语言模型能生成连贯长文，适合研究模型原理者阅读。
为何现代语言模型在训练时只预测下一个词，却能生成连贯文档并捕捉长距离结构？本文证明，即使使用常见的神经网络架构，优化下一个词预测也能有效学习更长距离的结构。具体而言，我们证明：在循环神经网络（RNN）上优化下一个词预测，可使模型对训练分布产生良好近似——对于从训练分布中抽取的保留文档，任何描述长度有限、仅观察接下来k个词的算法，都无法区分该文档中的连续k个词与模型在相同前缀下生成的k个词。我们给出了实现这种k-词不可区分性的模型规模的多项式界（关于k，与文档长度无关），为实际中观察到的长距离连贯性提供了复杂性理论解释。
原文摘要 · Abstract (English)
Why do modern language models, trained to do well on next-word prediction, appear to generate coherent documents and capture long-range structure? Here we show that next-token prediction is provably powerful for learning longer-range structure, even with common neural network architectures. Specifically, we prove that optimizing next-token prediction over a Recurrent Neural Network (RNN) yields a model that closely approximates the training distribution: for held-out documents sampled from the training distribution, no algorithm of bounded description length limited to examining the next $k$ tokens, for any $k$, can distinguish between $k$ consecutive tokens of such documents and $k$ tokens generated by the learned language model following the same prefix. We provide polynomial bounds (in $k$, independent of the document length) on the model size needed to achieve such $k$-token indistinguishability, offering a complexity-theoretic explanation for the long-range coherence observed in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。