用规则强化学习让大模型学会严谨推理，5千题训练后搞定高难度数学竞赛
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
- 用逻辑谜题+规则奖励机制引导模型逐步思考
- 70亿参数模型在5000题训练后通过AIME/AMC测试
- 适合想提升模型推理能力的研究者和工程师
受DeepSeek-R1成功的启发，我们探索了基于规则的强化学习（Rule-Based RL）在大型推理模型中的潜力。为分析推理动态，我们采用可调控复杂度且答案验证简单的合成逻辑谜题作为训练数据。提出了几项关键技术贡献：强调思维与作答过程的系统提示、严格格式的奖励函数（惩罚走捷径行为）、以及能稳定收敛的简单训练方案。我们的7B模型发展出反思、验证、总结等原始语料中不存在的高级推理技能。值得注意的是，仅在5000个逻辑问题上训练后，该模型便展现出对AIME和AMC等高难度数学基准的泛化能力。
原文摘要 · Abstract (English)
Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in large reasoning models. To analyze reasoning dynamics, we use synthetic logic puzzles as training data due to their controllable complexity and straightforward answer verification. We make some key technical contributions that lead to effective and stable RL training: a system prompt that emphasizes the thinking and answering process, a stringent format reward function that penalizes outputs for taking shortcuts, and a straightforward training recipe that achieves stable convergence. Our 7B model develops advanced reasoning skills-such as reflection, verification, and summarization-that are absent from the logic corpus. Remarkably, after training on just 5K logic problems, it demonstrates generalization abilities to the challenging math benchmarks AIME and AMC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。