用概念预测提升语言模型，训练更高效。
Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
- 用向量量化将隐藏状态转为离散概念，预测跨多个词的概念
- 在13个基准上表现优于传统分词模型，大模型下提升显著
- 适合追求更强语言建模能力的研究者与工程师
我们提出一种基于下一项词预测（NTP）的生成预训练范式——下一项概念预测（NCP），通过预测跨越多个词的离散概念，构建更具挑战性的预训练目标。ConceptLM 模型利用向量量化对隐藏状态进行离散化，建立概念词汇表，并同时使用 NCP 与 NTP 进行参数更新，以概念引导后续词的生成。我们在 70M 到 1.5B 参数规模的模型上从头训练，使用最多达 3000 亿词的训练数据，涵盖 Pythia 与 GPT-2 架构。在 13 个基准测试中，NCP 表现持续优于传统分词级模型。此外，在 80 亿参数的 Llama 模型上进行持续预训练的结果显示，NCP 可进一步提升已通过 NTP 训练的模型性能。分析表明，通过引入更难的预训练任务，NCP 能有效提升语言模型能力，为更好建模提供可行路径。
原文摘要 · Abstract (English)
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。