通过预测上下文提升语言模型效率，用更少参数达到更好效果。
Context-level Language Modeling by Learning Predictive Context Embeddings
- 用多标记上下文嵌入替代单标记预测，隐式学习长程依赖
- 参数减少39%即可达基线困惑度，训练成本更低
- 适合追求高效模型的开发者，尤其在资源受限场景
我们提出ContextLM，一种通过引入内在的下一段落预测目标来增强标准预训练的框架，使语言模型基于跨多个词的上下文嵌入进行构建，从而通过预测下一个上下文实现更好的下一个词预测。该模型完全兼容标准自回归、逐词评估范式（如困惑度）。在GPT-2和Pythia主干网络上的大量实验（最大15亿参数，3000亿训练词）表明，ContextLM将缩放定律的帕累托前沿向更高效方向移动，在参数量、训练词数和浮点运算次数方面均表现更优。结果表明，ContextLM在参数量减少39%的情况下即可达到基线困惑度，并在同等参数量下对广泛下游任务表现出更强的泛化能力。
原文摘要 · Abstract (English)
We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39\% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。