用强化学习训练语言模型预测下一个词，提升准确性。
Reinforcement Pre-Training
- 将文本预测任务转为可验证奖励的推理问题，用强化学习训练。
- 训练算力越大，预测准确率越高，表现随规模持续提升。
- 适合需要通用强化学习基础的语言模型预训练场景。
本文提出强化预训练（RPT），作为大语言模型与强化学习（RL）的一种新型扩展范式。具体而言，将下一个词预测重构为一个可通过强化学习训练的推理任务，模型在正确预测给定上下文的下一个词时获得可验证奖励。RPT提供了一种可扩展的方法，利用海量文本数据实现通用强化学习，而非依赖特定领域标注答案。通过激励下一个词推理能力，RPT显著提升了语言建模中对下一个词的预测准确性。此外，RPT为后续强化微调提供了强大的预训练基础。扩展曲线显示，随着训练算力增加，下一个词预测准确率持续提升。结果表明，RPT是一种有效且有前景的扩展范式，可推动语言模型预训练发展。
原文摘要 · Abstract (English)
In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。