GRPO其实暗含对比学习机制，两样本即可高效训练大模型。
It Takes Two: Your GRPO Is Secretly DPO
- 用两个样本构造对比信号，替代传统大组统计
- 性能达16人组的97.6%，仅需12.5%的推演量
- 适合资源有限但追求高效对齐的LLM训练场景
GRPO作为后训练大模型的主流强化学习算法，通过群体级统计估计价值基线，无需评论家网络。当前普遍认为需要大组规模才能获得准确估计。本文提出新视角：GRPO的有效性源于其优化中隐含的对比目标，通过控制变量法降低方差，使其在结构上与偏好学习方法（如DPO）相关。基于此，我们提出2-GRPO，仅需两个采样轨迹即可构建对比信号。理论分析和实验验证表明，2-GRPO保留了16-GRPO 97.6%的性能，同时仅需12.5%的推演次数和21%的训练时间。
原文摘要 · Abstract (English)
GRPO has emerged as a prominent reinforcement learning algorithm for post-training LLMs. Unlike critic-based methods, GRPO computes advantages by estimating the \emph{value baselines} from group-level statistics, eliminating the need for a critic network. Consequently, the prevailing view emphasizes the necessity of large group sizes, which are assumed to yield more accurate statistical estimates. In this paper, we propose a different view that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This makes GRPO structurally related to preference learning methods such as DPO. This perspective motivates 2-GRPO, a minimal group-size variant that constructs contrastive signals with only two rollouts. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains $97.6\%$ of the performance of 16-GRPO, while requiring only $12.5\%$ of the rollouts and $21\%$ of the training time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。