让大模型预测下一个词的隐含语义，提升泛化能力。
NITP: Next Implicit Token Prediction for LLM Pre-training
- 用模型自身浅层表示作为目标，对隐含语义进行连续监督。
- 90亿参数MoE模型在MMLU-Pro上提升5.7%，额外计算仅增2%。
- 适合追求性能提升且不增加推理成本的LLM训练场景。
标准的下一个词预测（NTP）仅在输出逻辑空间使用离散标签进行监督，这种稀疏的独热标签使潜在表示空间约束不足，导致隐藏状态可能漂移至退化且各向异性的配置，限制泛化能力。为此，我们提出下一代隐含词预测（NITP），通过在表示空间中引入密集连续监督，增强离散预测。NITP训练模型预测下一个词的隐含语义内容，采用同一模型浅层表示作为稳定自监督目标。理论分析表明，NITP通过缓解自由度不足问题，促进紧凑、结构化的表示几何。实验上，从0.5B到9B参数的密集与MoE模型均显示一致提升，计算开销极低。在9B MoE模型上，NITP在MMLU-Pro上实现5.7%绝对提升，同时在C3和CommonsenseQA上分别提升6.4%和4.3%，训练额外消耗约2% FLOPs，无额外推理成本。
原文摘要 · Abstract (English)
Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that can limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense continuous supervision directly in the representation space. NITP trains the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. We provide theoretical analysis showing that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and encouraging a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. On a 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with approximately 2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。