开源实现大模型强化学习训练，用简单方法达到顶尖性能。
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
- 仅用基础PPO+规则奖励，无KL惩罚，实现高效训练。
- 在多个基准上超越原版模型，训练步数减少90%。
- 适合关注可复现性与高效强化学习的开发者与研究者。
我们提出Open-Reasoner-Zero，首个面向大规模推理型强化学习训练的开源实现，聚焦可扩展性、简洁性与易用性。通过大量实验表明，仅使用基础PPO结合GAE（λ=1, γ=1）和直接规则奖励，不加KL正则化，即可实现性能与响应长度的规模化提升，复现DeepSeek-R1-Zero的缩放现象。采用与DeepSeek-R1-Zero-Qwen-32B相同的Qwen2.5-32B基础模型，在AIME2024、MATH500和GPQA Diamond三个基准上均表现更优，且训练步骤仅为原流程的1/10。分析显示，训练中学习到的评价器能有效识别并削弱重复回应模式，提升优势估计稳定性。我们开源代码、训练数据及多种模型权重，推动可复现性与相关模型探索。
原文摘要 · Abstract (English)
We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training on the base model focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE ($λ=1$, $γ=1$) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both benchmark performance and response length, replicating the scaling phenomenon observed in DeepSeek-R1-Zero. Using the same base model, Qwen2.5-32B base, as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance across AIME2024, MATH500, and GPQA Diamond, while demonstrating remarkable efficiency, requiring only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline. Moreover, our analysis not only covers training dynamics and ablation for critical design choices, but also quantitatively shows how the learned critic in Reasoner-Zero training effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stability. Embracing the principles of open-source, we release our source code, training data, and various model weights, fostering reproducibility and encouraging further exploration of the properties of related models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。