揭示大模型强化学习训练中的隐性偏差，提出更高效的优化方法
Understanding R1-Zero-Like Training: A Critical Perspective
- 分析多种基础模型在强化学习中的表现，发现预训练阶段已存在推理优势
- 发现GRPO优化器会人为拉长错误回答长度，导致效率下降
- 提出新优化方法Dr. GRPO，用7B模型在AIME 2024上达43.3%准确率
DeepSeek-R1-Zero表明大规模强化学习可直接提升大模型的推理能力，无需监督微调。本文从基础模型和强化学习两方面深入分析这类训练范式。我们考察了包括DeepSeek-V3-Base在内的多种基础模型，发现其在预训练阶段即表现出'顿悟'现象，而Qwen2.5基座模型即使无提示模板也具备强推理能力，暗示预训练中存在潜在偏见。此外，我们识别出组相对策略优化（GRPO）存在优化偏差，会人为增加错误输出的响应长度。为此，我们提出无偏优化方法Dr. GRPO，显著提升词元效率并保持推理性能。基于此，我们构建了一个极简的R1-Zero训练方案，在7B基座模型上实现AIME 2024 43.3%的准确率，创下新纪录。代码已开源。
原文摘要 · Abstract (English)
DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。