NTP训练比专注答案的CTP更能提升模型推理能力。
Reasoning Bias of Next Token Prediction Training
- 用全词预测（NTP）训练，而非只预测关键答案
- NTP模型在多个推理数据集上表现更好，更抗干扰
- 适合预训练阶段使用，能增强模型泛化能力
自大型语言模型（LLMs）诞生以来，如何高效训练以获得优越推理能力始终是核心挑战。当前主流训练范式基于下一个词预测（NTP），而另一种方法称为关键词预测（CTP），仅关注特定关键标记（如问答数据集中的答案），旨在减少冗余信息和噪声的过拟合。然而，我们的研究发现，尽管NTP在训练中暴露于噪声，其推理能力反而优于CTP。我们将其归因于噪声对训练动态的正则化作用。实证分析表明，NTP训练的模型在多个基准推理数据集上展现出更强的泛化性和鲁棒性，对扰动更具抵抗力，并达到更平坦的损失极小值。这些发现揭示：NTP在预训练阶段对推理能力的培养至关重要，而CTP更适合微调阶段，从而深化了对大模型训练策略的科学认知。
原文摘要 · Abstract (English)
Since the inception of Large Language Models (LLMs), the quest to efficiently train them for superior reasoning capabilities has been a pivotal challenge. The dominant training paradigm for LLMs is based on next token prediction (NTP). Alternative methodologies, called Critical Token Prediction (CTP), focused exclusively on specific critical tokens (such as the answer in Q\&A dataset), aiming to reduce the overfitting of extraneous information and noise. Contrary to initial assumptions, our research reveals that despite NTP's exposure to noise during training, it surpasses CTP in reasoning ability. We attribute this counterintuitive outcome to the regularizing influence of noise on the training dynamics. Our empirical analysis shows that NTP-trained models exhibit enhanced generalization and robustness across various benchmark reasoning datasets, demonstrating greater resilience to perturbations and achieving flatter loss minima. These findings illuminate that NTP is instrumental in fostering reasoning abilities during pretraining, whereas CTP is more effective for finetuning, thereby enriching our comprehension of optimal training strategies in LLM development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。