用简单方法提升大模型推理能力，效果超越现有强化学习方案。
GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
- 直接优化原始强化学习目标，省去复杂辅助模块。
- 在多模态任务中性能优于GRPO，且计算开销更低。
- 适合追求高效、稳定推理增强的开发者和研究者。
强化学习（RL）可在不依赖大量监督微调（SFT）的情况下直接提升大语言模型的推理能力。本文重新审视传统的策略梯度（PG）机制，提出一种极简的强化学习方法——组策略梯度（GPG）。与传统方法不同，GPG直接优化原始RL目标，无需代理损失函数。通过移除价值网络和参考模型，避免KL散度约束，并解决优势估计与梯度偏差问题，训练过程显著简化，相比组相对策略优化（GRPO）更加高效。实验表明，该方法在多种单模态与多模态任务中均表现更优，且计算成本更低。代码已开源：https://github.com/AMAP-ML/GPG。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) can directly enhance the reasoning capabilities of large language models without extensive reliance on Supervised Fine-Tuning (SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimize the original RL objective, thus obviating the need for surrogate loss functions. By eliminating the critic and reference models, avoiding KL divergence constraints, and addressing the advantage and gradient estimation bias, our approach significantly simplifies the training process compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. As illustrated in Figure 1, extensive experiments demonstrate that our method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks. Our code is available at https://github.com/AMAP-ML/GPG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。