通过分层强化学习提升大模型推理与自纠错能力
Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
- 采用双层GRPO结构，先生成答案再自动修正错误
- 在数学推理任务上超越标准GRPO，显著提升准确率
- 无需密集标注奖励模型，实现隐式过程监督
Group Relative Policy Optimization（GRPO）算法在提升大语言模型推理能力方面表现优异，如DeepSeek-R1所示。然而，由于缺乏中间环节的监督，GRPO常导致探索效率低下。复杂推理链中一旦出现错误，整个解题过程即失效，造成奖励骤降，影响训练稳定性。为此，我们提出MGRPO（多层GRPO）。该方法分为两层：第一层使用标准GRPO生成初始回答；随后将原始问题与初始回答输入第二层GRPO，专门用于识别并纠正初始回答中的错误，形成自纠错循环。该机制通过奖励成功纠错行为，实现隐式过程级监督，无需依赖显式的密集标注奖励模型。在多个数学推理基准测试中，MGRPO显著优于标准GRPO，有效增强模型的推理与自纠错能力。
原文摘要 · Abstract (English)
The Group Relative Policy Optimization (GRPO) algorithm has demonstrated considerable success in enhancing the reasoning capabilities of large language models (LLMs), as evidenced by DeepSeek-R1. However, the absence of intermediate supervision in GRPO frequently leads to inefficient exploration dynamics. A single error in a complex reasoning chain can invalidate the entire solution, resulting in abrupt reward vanishing and compromising training stability.To address these challenges, we propose MGRPO (Multi-layer GRPO). MGRPO operates in two layers: the first layer employs standard GRPO to generate an initial response. This response, along with the original query, is then fed into a second-layer GRPO process. This second layer is specifically trained to identify and correct errors in the initial response, effectively creating a self-correction loop. This mechanism provides implicit process-level supervision by rewarding successful error correction, without requiring an explicit, densely-annotated reward model. Experimental results on several mathematical reasoning benchmarks demonstrate that MGRPO significantly outperforms standard GRPO, achieving superior performance by fostering both reasoning and self-correction abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。