简化强化学习方法也能有效提升大模型推理能力
Are complicated loss functions necessary for teaching LLMs to reason?
- 用群体相对优势估计替代复杂惩罚项，降低训练难度
- 去掉PPO式剪裁后数学推理性能仍优于原方法
- 适合追求高效透明训练方案的研究者使用
近期大语言模型在推理与数学能力方面的进展凸显了后训练技术的重要性。群组相对策略优化（GRPO）通过结合群组相对优势估计、类似PPO的剪裁机制和KL正则化展现潜力。然而其复杂性引发疑问：是否所有组件均必要？我们系统分析了GRPO，发现两点关键：(1) 引入负反馈至关重要，仅训练高于基线的行为会限制学习；(2) 类似PPO的约束（如策略比率剪裁）并非提升数学推理所必需。基于此，我们提出简化版本RGRA，保留群组相对优势估计，但移除PPO剪裁与策略比率项。在标准数学基准测试中，RGRA表现优于GRPO，表明基于REINFORCE的简化方法可有效增强大模型推理能力，提供更透明高效的替代方案。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) highlight the importance of post training techniques for improving reasoning and mathematical ability. Group Relative Policy Optimization (GRPO) has shown promise in this domain by combining group relative advantage estimation, PPO style clipping, and KL regularization. However, its complexity raises the question of whether all components are necessary for fostering reasoning behaviors. We conduct a systematic analysis of GRPO and identify two key findings: (1) incorporating negative feedback is essential training solely on actions above a baseline limits learning; and (2) PPO style constraints, such as policy ratio clipping, are not required to improve mathematical reasoning or performance. Building on these insights, we propose REINFORCE with Group Relative Advantage (RGRA), a simplified variant that retains group relative advantage estimation but removes PPO style clipping and policy ratio terms. Experiments across standard mathematical benchmarks indicate that RGRA has the potential to achieve stronger performance than GRPO. Our results suggest that simpler REINFORCE based approaches can effectively enhance reasoning in LLMs, offering a more transparent and efficient alternative to GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。