用未来摘要替代传统预测，让大模型更擅长长程推理与创作。
Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
- 引入未来摘要预测，通过压缩未来内容提升长序列生成能力
- 3B和8B模型在数学、推理、编码任务上优于传统NTP和MTP
- 适合需要长文本规划与创造性写作的研究者使用
下一代语言模型的成功依赖于下一词预测（NTP），但其在长程推理、规划和创意写作方面表现受限，主要源于教师强制训练。多词预测（MTP）虽部分缓解问题，但仍局限于短距离依赖，提升有限。本文提出未来摘要预测（FSP），通过辅助头预测未来内容的紧凑表示，保留长时生成所需信息。探索两种变体：人工设计摘要（如未来序列的词袋表示）和学习型摘要（由从右到左训练的反向语言模型生成嵌入）。大规模预训练实验（3B和8B参数模型）表明，FSP在数学、推理和编码基准上均优于NTP和MTP。
原文摘要 · Abstract (English)
Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by predicting several future tokens at once, but it mostly captures short-range dependencies and offers limited improvement. We propose future summary prediction (FSP), which trains an auxiliary head to predict a compact representation of the long-term future, preserving information relevant for long-form generations. We explore two variants of FSP: handcrafted summaries, for example, a bag of words summary of the future sequence, and learned summaries, which use embeddings produced by a reverse language model trained from right-to-left order. Large-scale pretraining experiments (3B and 8B-parameter models) demonstrate that FSP provides improvements over both NTP and MTP across math, reasoning, and coding benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。