arXiv:2505.16400cs.LGcs.AI2025-05NeurIPS被引 73

用强化学习提升小模型的数学与编程推理能力,效果超越当前主流蒸馏方法。

AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning

  • 先训数学、再训代码,分阶段强化学习提升推理能力
  • 7B/14B模型在AIME2025上提升14.6%/17.2%,代码任务也显著进步
  • 适合想用小模型实现强推理的开发者和研究者

尽管大规模强化学习(RL)在推理任务上取得进展,但高性能推理模型的训练方法仍不明确。前沿模型如DeepSeek-R1的关键实现细节常被省略。近期研究表明,对小型模型而言,蒸馏仍优于强化学习。本文证明,大规模强化学习可显著提升中小型模型的推理能力,超越现有蒸馏模型。通过系统性消融实验,提出简单有效的方法:先在纯数学提示上训练,再在纯代码提示上训练。结果显示,仅数学强化学习即能显著提升强蒸馏模型在数学基准上的表现(如7B/14B模型在AIME 2025上分别提升+14.6% / +17.2%),并同时改善代码推理能力(如7B/14B模型在LiveCodeBench上分别提升+6.8% / +5.8%)。进一步延长代码强化学习迭代,可提升代码性能且不影响数学表现。我们构建了高质量数据管道,收集具有可验证答案和测试用例的挑战性提示,支持跨领域的验证式强化学习。还发现课程学习(逐步增加响应长度)和在线策略参数更新具有稳定作用。强化学习不仅能激发预训练与监督微调中已习得的基础推理能力,还能突破模型原有极限,解决此前无法求解的问题。

原文摘要 · Abstract (English)

Despite recent progress in large-scale reinforcement learning (RL) for reasoning, the training recipe for building high-performing reasoning models remains elusive. Key implementation details of frontier models, such as DeepSeek-R1, including data curation strategies and RL training recipe, are often omitted. Moreover, recent research indicates distillation remains more effective than RL for smaller models. In this work, we demonstrate that large-scale RL can significantly enhance the reasoning capabilities of strong, small- and mid-sized models, achieving results that surpass those of state-of-the-art distillation-based models. We systematically study the RL training process through extensive ablations and propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts. Notably, we find that math-only RL not only significantly enhances the performance of strong distilled models on math benchmarks (e.g., +14.6% / +17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks (e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition, extended code-only RL iterations further improve performance on code benchmarks with minimal or no degradation in math results. We develop a robust data curation pipeline to collect challenging prompts with high-quality, verifiable answers and test cases to enable verification-based RL across both domains. Finally, we identify key experimental insights, including curriculum learning with progressively increasing response lengths and the stabilizing effect of on-policy parameter updates. We find that RL not only elicits the foundational reasoning capabilities acquired during pretraining and supervised fine-tuning (e.g., distillation), but also pushes the limits of the model's reasoning ability, enabling it to solve problems that were previously unsolvable.

强化学习数学推理代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。