解决大模型强化学习中奖励趋同问题，提升推理多样性。
EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
- 基于样本级熵驱动优势计算，避免奖励坍塌。
- 引入引导式错误修正机制，增强训练信号。
- 适用于需要多样化推理的复杂任务场景。
大型语言模型（LLMs）通过强化学习在逐步推理方面取得了显著进展。然而，依赖稀疏奖励规则的分组相对策略优化（GRPO）算法常因组内奖励相同导致优势坍塌问题。现有方法主要从模型自我反思以增强响应多样性、引入内部反馈以丰富训练信号（优势）两方面着手。本文首先分析模型反思的局限性，并研究了细粒度样本层面的策略熵。基于实验发现，提出EDGE-GRPO算法，采用熵驱动优势与引导式错误修正，有效缓解优势坍塌问题。在多个主流推理基准上的大量实验表明该方法具有有效性与优越性。代码已开源：https://github.com/ZhangXJ199/EDGE-GRPO。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have made remarkable progress in enhancing step-by-step reasoning through reinforcement learning. However, the Group Relative Policy Optimization (GRPO) algorithm, which relies on sparse reward rules, often encounters the issue of identical rewards within groups, leading to the advantage collapse problem. Existing works typically address this challenge from two perspectives: enforcing model reflection to enhance response diversity, and introducing internal feedback to augment the training signal (advantage). In this work, we begin by analyzing the limitations of model reflection and investigating the policy entropy of responses at the fine-grained sample level. Based on our experimental findings, we propose the EDGE-GRPO algorithm, which adopts \textbf{E}ntropy-\textbf{D}riven Advantage and \textbf{G}uided \textbf{E}rror Correction to effectively mitigate the problem of advantage collapse. Extensive experiments on several main reasoning benchmarks demonstrate the effectiveness and superiority of our approach. It is available at https://github.com/ZhangXJ199/EDGE-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。