arXiv:2507.08267cs.LGcs.AI2025-07

先用长周期微调提准确率,再用强化学习优化解题效率。

A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning

  • 分两阶段训练:先做10轮以上监督微调,再用GRPO优化推理效率
  • 在数学奥赛中排名前1%(超2200支队伍),解题准确率突破极限
  • 适合追求高精度且需高效推理的数学AI研究者与应用开发者

提升大语言模型的数学推理能力是推动AI发展的重要挑战。尽管监督微调(SFT)和强化学习(RL)是主流训练范式,但如何系统结合二者以同时最大化准确率与效率仍缺乏有效方法。本文提出一种实用的两阶段训练方案,将延长的SFT与在线推理中的强化学习(GRPO)有机结合。我们认为二者互补而非冲突:先通过长达10轮的SFT将模型准确率推至极限,随后通过GRPO显著提升生成效率并保持最优性能。实验表明,延长SFT至10轮是实现性能跃升的关键,而GRPO的核心作用在于压缩解题长度。该方法在多个严苛基准上表现卓越,尤其在无泄露数据的AI数学奥林匹克竞赛(AIMO)中位列全球前1%(超2200支队伍)。为保障可复现性,作者将开源完整框架,包括代码、模型权重与训练配置,详见https://github.com/analokmaus/kaggle-aimo2-fast-math-r1。

原文摘要 · Abstract (English)

Enhancing the mathematical reasoning of Large Language Models (LLMs) is a pivotal challenge in advancing AI capabilities. While Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are the dominant training paradigms, a systematic methodology for combining them to maximize both accuracy and efficiency remains largely unexplored. This paper introduces a practical and effective training recipe that strategically integrates extended SFT with RL from online inference (GRPO). We posit that these methods play complementary, not competing, roles: a prolonged SFT phase first pushes the model's accuracy to its limits, after which a GRPO phase dramatically improves token efficiency while preserving this peak performance. Our experiments reveal that extending SFT for as many as 10 epochs is crucial for performance breakthroughs, and that the primary role of GRPO in this framework is to optimize solution length. The efficacy of our recipe is rigorously validated through top-tier performance on challenging benchmarks, including a high rank among over 2,200 teams in the strictly leak-free AI Mathematical Olympiad (AIMO). This work provides the community with a battle-tested blueprint for developing state-of-the-art mathematical reasoners that are both exceptionally accurate and practically efficient. To ensure full reproducibility and empower future research, we will open-source our entire framework, including all code, model checkpoints, and training configurations at https://github.com/analokmaus/kaggle-aimo2-fast-math-r1.

数学推理两阶段训练强化学习高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。