通过自反馈迭代优化,让大模型在数学推理上更准更稳。
iGRPO: Self-Feedback-Driven LLM Reasoning
- 用自我生成草稿动态调整策略,分两阶段提升推理质量。
- 在AIME24/25上达到85.62%和79.64%新高,超越基线模型。
- 适合追求高精度数学推理的开发者与研究者使用。
大型语言模型在解决复杂数学问题方面展现出潜力,但仍难以保证结果的准确性和一致性。强化学习(RL)通过任务特定奖励对齐模型,提升整体质量与可靠性。组相对策略优化(GRPO)是一种高效、无需价值函数的近端策略优化(PPO)替代方法,利用组相对奖励归一化。本文提出迭代组相对策略优化(iGRPO),是GRPO的两阶段扩展,通过模型生成的草稿实现动态自条件化。第一阶段,采样多个探索性草稿,并用相同的标量奖励信号选择最高奖励草稿;第二阶段,将此最优草稿附加至原始提示,对草稿条件化修正应用类似GRPO的更新,训练策略在最强先前尝试基础上进一步改进。在相同回溯预算下,iGRPO在多个基础模型(如Nemotron-H-8B-Base-8K和DeepSeek-R1 Distilled)上持续优于GRPO。此外,将iGRPO应用于在AceReason-Math上训练的OpenReasoning-Nemotron-7B,在AIME24和AIME25上分别取得85.62%和79.64%的新最优成绩。消融实验表明,该修正框架不仅适用于GRPO变体,还能从生成式裁判中获益,并通过延迟熵崩溃改变学习动态。这些结果凸显了迭代自反馈强化学习在可验证数学推理中的潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions. Reinforcement Learning (RL) is a framework for aligning these models with task-specific rewards, improving overall quality and reliability. Group Relative Policy Optimization (GRPO) is an efficient, value-function-free alternative to Proximal Policy Optimization (PPO) that leverages group-relative reward normalization. We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts. In Stage 1, iGRPO samples multiple exploratory drafts and selects the highest-reward draft using the same scalar reward signal used for optimization. In Stage 2, it appends this best draft to the original prompt and applies a GRPO-style update on draft-conditioned refinements, training the policy to improve beyond its strongest prior attempt. Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models (e.g., Nemotron-H-8B-Base-8K and DeepSeek-R1 Distilled), validating its effectiveness on diverse reasoning benchmarks. Moreover, applying iGRPO to OpenReasoning-Nemotron-7B trained on AceReason-Math achieves new state-of-the-art results of 85.62\% and 79.64\% on AIME24 and AIME25, respectively. Ablations further show that the refinement wrapper generalizes beyond GRPO variants, benefits from a generative judge, and alters learning dynamics by delaying entropy collapse. These results underscore the potential of iterative, self-feedback-based RL for advancing verifiable mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。