让大模型通过自我纠错提升数学推理能力
ScRPO: From Errors to Insights
- 分两阶段训练：先试错积累错误，再反思修正逻辑漏洞
- 在AIME等数据集上准确率提升6%~7%，超越现有方法
- 适合需要自主改进的AI系统，尤其缺乏外部反馈场景
我们提出自校正相对策略优化（ScRPO），一种新型强化学习框架，通过迭代自我反思与错误修正，提升大语言模型的高级数学推理能力。该框架分为两个阶段：(1) 试错学习阶段，模型通过GRPO训练，收集错误回答形成“错误池”；(2) 自我校正阶段，引导模型内省分析并修正先前错误背后的推理缺陷。在AIME、AMC、奥数、MATH-500和GSM8k等挑战性数学基准上的评估验证了该方法的有效性。以DeepSeek-R1-Distill-Qwen-1.5B和7B为基座模型，ScRPO分别达到64.8%和77.8%的平均准确率，较原始基线分别提升6.0%和3.2%，持续优于DAPO和GRPO等强后训练方法。这些结果确立了ScRPO作为实现AI系统自主持续改进的稳健范式，尤其适用于外部反馈有限的任务。
原文摘要 · Abstract (English)
We introduce Self-correction Relative Policy Optimization (ScRPO), a novel reinforcement learning framework designed to empower large language models with advanced mathematical reasoning capabilities through iterative self-reflection and error correction. The ScRPO framework operates in two distinct phases: (1) Trial-and-error learning stage, where the model is trained via GRPO, and incorrect responses are collected to form an "error pool"; and (2) Self-correction learning stage, which guides the model to introspectively analyze and rectify the reasoning flaws behind its previous errors. Extensive evaluations across challenging mathematical benchmarks, including AIME, AMC, Olympiad, MATH-500, and GSM8k, validate the efficacy of our approach. Using DeepSeek-R1-Distill-Qwen-1.5B and 7B as backbones, ScRPO achieves average accuracies of 64.8% and 77.8%, respectively. This represents a significant improvement of 6.0% and 3.2% over vanilla baselines, consistently outperforming strong post-training methods such as DAPO and GRPO. These findings establish ScRPO as a robust paradigm for enabling autonomous self-improvement in AI systems, particularly in tasks with limited external feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。