用强化学习提升大模型的慢思考能力，显著提高数学推理准确率。
An Empirical Study on Eliciting and Improving R1-like Reasoning Models
- 通过强化学习优化基础模型与微调模型的推理能力。
- 在AIME 2024上达到86.67%准确率，远超原有水平。
- 适合研究大模型推理机制与数学问题求解的开发者。
本报告是STILL项目中关于慢思考模型开发的第三份技术报告。随着技术路径日益清晰，强化学习（RL）训练已成为构建此类推理模型的核心方法。我们系统性地实验并记录了影响RL训练的多种因素，涵盖基础模型与微调模型。结果表明，我们的RL训练方法持续提升了Qwen2.5-32B基础模型的响应长度与测试准确率。此外，即使像DeepSeek-R1-Distill-Qwen-1.5B这样已具备高性能的模型，仍可通过RL训练进一步优化，在AIME 2024上达到39.33%的准确率。除RL外，我们还探索了工具操作的应用，发现其显著增强大模型的推理表现：采用贪婪搜索时，在AIME 2024上实现86.67%的准确率，充分验证了该方法的有效性。相关资源已在STILL项目官网公开：https://github.com/RUCAIBox/Slow_Thinking_with_LLMs。
原文摘要 · Abstract (English)
In this report, we present the third technical report on the development of slow-thinking models as part of the STILL project. As the technical pathway becomes clearer, scaling RL training has become a central technique for implementing such reasoning models. We systematically experiment with and document the effects of various factors influencing RL training, conducting experiments on both base models and fine-tuned models. Specifically, we demonstrate that our RL training approach consistently improves the Qwen2.5-32B base models, enhancing both response length and test accuracy. Furthermore, we show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already achieved a high performance level, it can be further refined through RL training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we also explore the use of tool manipulation, finding that it significantly boosts the reasoning performance of large reasoning models. This approach achieves a remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its effectiveness in enhancing model capabilities. We release our resources at the STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。