让小模型通过长期强化学习实现多领域推理能力飞跃
Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
- 用长期强化学习+可验证奖励信号训练小模型，提升推理能力
- 数学、编程、逻辑题分别提升14.7%、13.9%、54.8%
- 适合想提升模型推理能力的研究者和开发者
近期以OpenAI的O1和DeepSeek-R1为代表的推理型语言模型表明，通过链式思维和迭代探索等测试时计算扩展，可在数学与代码生成等复杂任务上取得显著进步。这些突破主要得益于大规模强化学习，尤其是结合可验证奖励信号提供的客观、可靠监督。本文研究了在小型语言模型上进行长期强化学习的效果，覆盖多个推理领域。我们识别出有效训练的关键要素：使用可验证奖励任务、改进组相对策略优化（GRPO），以及提升训练稳定性和泛化性的实用技术。引入受控KL正则化、比例截断和周期性参考策略重置作为长期性能提升的核心组件。模型在强基线基础上实现显著提升：数学任务+14.7%，编程任务+13.9%，逻辑谜题任务+54.8%。为促进后续研究，我们公开发布该模型。
原文摘要 · Abstract (English)
Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。