让语言模型提前规划未来文本，提升预测准确率。
Learning to Plan Long-Term for Language Modeling
- 通过采样多个未来文本计划，增强模型对后续内容的预判能力。
- 在WikiText-103数据集上，模型生成质量提升，困惑度降低1.8点。
- 适合需要高精度长文本生成的任务，如写作辅助与内容创作。
现代语言模型通过注意力机制基于历史文本预测下一个词元，但缺乏显式机制来分配计算资源以规划远距离未来文本，导致次优的词元预测。本文提出一种规划器，可预测未来多句文本的潜在计划。通过一次性采样多个计划，使语言模型能够基于文本延续分布的准确近似进行条件化，从而提升下一个词元的预测准确性。该方法本质上实现了用计算时间换取预测精度的权衡。
原文摘要 · Abstract (English)
Modern language models predict the next token in the sequence by considering the past text through a powerful function such as attention. However, language models have no explicit mechanism that allows them to spend computation time for planning long-distance future text, leading to a suboptimal token prediction. In this paper, we propose a planner that predicts a latent plan for many sentences into the future. By sampling multiple plans at once, we condition the language model on an accurate approximation of the distribution of text continuations, which leads to better next token prediction accuracy. In effect, this allows trading computation time for prediction accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。