arXiv:2508.20722cs.CL2025-08被引 46

14B模型通过智能体强化学习,在有限算力下实现顶尖数学推理能力。

rStar2-Agent: Agentic Reasoning Technical Report

  • 用高效强化学习框架与代码环境,降低训练成本
  • 在AIME24和AIME25上分别达80.6%和69.8%准确率
  • 适合关注智能体推理、低资源训练的开发者

我们提出rStar2-Agent,一个140亿参数的数学推理模型,通过智能体强化学习实现前沿性能。该模型展现出复杂问题求解中的高级认知行为,如在调用Python工具前谨慎思考,并根据代码执行反馈自主探索、验证与优化中间步骤。这一能力得益于三项关键创新:(i)高效的强化学习基础设施,配备可靠的Python代码环境,支持高吞吐执行并降低采样开销,使模型能在64块MI300X GPU上训练;(ii)GRPO-RoC算法采用Resample-on-Correct策略,缓解代码工具带来的环境噪声,提升推理效率;(iii)高效的训练流程,从非推理微调开始,经多阶段强化学习,以极低计算成本激发高级认知能力。仅用510步强化学习,在一周内将预训练140亿参数模型推至顶尖水平,在AIME24上平均通过率为80.6%,AIME25为69.8%,超越6710亿参数的DeepSeek-R1,且响应更短。该模型在对齐、科学推理和工具使用任务中也展现强泛化能力。代码与训练方案见https://github.com/microsoft/rStar。

原文摘要 · Abstract (English)

We introduce rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning to achieve frontier-level performance. Beyond current long CoT, the model demonstrates advanced cognitive behaviors, such as thinking carefully before using Python coding tools and reflecting on code execution feedback to autonomously explore, verify, and refine intermediate steps in complex problem-solving. This capability is enabled through three key innovations that makes agentic RL effective at scale: (i) an efficient RL infrastructure with a reliable Python code environment that supports high-throughput execution and mitigates the high rollout costs, enabling training on limited GPU resources (64 MI300X GPUs); (ii) GRPO-RoC, an agentic RL algorithm with a Resample-on-Correct rollout strategy that addresses the inherent environment noises from coding tools, allowing the model to reason more effectively in a code environment; (iii) An efficient agent training recipe that starts with non-reasoning SFT and progresses through multi-RL stages, yielding advanced cognitive abilities with minimal compute cost. To this end, rStar2-Agent boosts a pre-trained 14B model to state of the art in only 510 RL steps within one week, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25, surpassing DeepSeek-R1 (671B) with significantly shorter responses. Beyond mathematics, rStar2-Agent-14B also demonstrates strong generalization to alignment, scientific reasoning, and agentic tool-use tasks. Code and training recipes are available at https://github.com/microsoft/rStar.

数学推理智能体强化学习低资源训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。