让大模型通过自然语言反馈改进推理，效果优于纯数值奖励方法。
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
- 结合自然语言批评与数值奖励，动态优化模型推理策略。
- 在多个任务上提升15%-21.6%的通过率，AIME 2024提升16.7%。
- 适合需要高阶逻辑推理与自我改进能力的研究与应用。
基于数值奖励的强化学习虽显著提升了大语言模型的复杂推理能力，但存在性能瓶颈、无效自发反思和持续失败三大局限。本文发现，在引入自然语言批评后，陷入瓶颈的强化学习模型可有效修正错误解法。为此提出Critique-GRPO，一种融合自然语言与数值反馈的在线强化学习框架，使模型能同时从初始回答和批判性修正中学习，充分吸收两阶段探索优势。大量实验表明，该方法在多个Qwen模型上实现约+15.0%-21.6%的Pass@1提升，对Llama-3.2-3B-Instruct提升+7.3%，并在八项挑战性推理任务中超越所有对比的监督与强化学习微调方法。尤其在自评自改场景下表现突出，相较GRPO在AIME 2024任务上取得+16.7%的显著增益。代码与模型已开源。
原文摘要 · Abstract (English)
Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs). However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures. We show that plateaued RL models can successfully refine failed solutions when given natural language critiques. Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization. This approach enables LLMs to learn simultaneously from initial responses and critique-guided refinements, effectively internalizing the exploration benefits of both stages. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024. The code and models are released at: https://github.com/zhangxy-2019/critique-GRPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。