用预测未来词序替代逐词预测，提升语言模型效果
Predicting the Order of Upcoming Tokens Improves Language Modeling
- 让模型预测未来多个词的相对顺序，使用排序损失训练
- 在9个标准任务上优于传统预测方法，70亿参数模型也表现更好
- 特别适合数学和代码任务，能解决其他方法失败的图路径问题
多词预测（MTP）作为辅助目标用于改进语言模型的下一步词预测（NTP），但在标准NLP基准测试中表现不一，常表现不佳。我们发现，精确预测未来词本身过于困难。因此提出词序预测（TOP），通过学习排序损失训练模型预测未来词的相对顺序。相比MTP需要额外多个Transformer层，TOP仅需一个额外的未嵌入层。我们在340M、1.8B和7B参数规模的模型上，分别用NTP、MTP、DeepSeek MTP（DS-MTP）和TOP目标进行预训练。在9个标准NLP基准测试中，TOP整体优于NTP、MTP和DS-MTP，即使在大规模模型下仍保持优势。继续在数学与代码数据上训练后，TOP模型在4个相关任务上表现更优。在合成星形图任务中，仅TOP能成功实现路径查找，而NTP、MTP和DS-MTP均失败。代码已开源。
原文摘要 · Abstract (English)
Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We found MTP's exact future token prediction to be too difficult as an auxiliary loss. Instead, we propose token order prediction (TOP), which trains models to order upcoming tokens by their proximity using a learning-to-rank loss. TOP requires only a single additional unembedding layer compared to MTP's multiple transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using NTP, MTP, DeepSeek MTP (DS-MTP) and TOP objectives. The results of nine standard NLP benchmarks show that TOP overall outperforms NTP, MTP, and DS-MTP even at scale. TOP models with continued training on math and code also perform better on 4 relevant benchmarks. On the synthetic star graph task, TOP enables pathfinding on graphs where NTP, MTP, and DS-MTP fail. Our code is available at https://github.com/zaydzuhri/token-order-prediction
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。