GVPO提升大模型后训练稳定性，解决奖励优化中的收敛难题。
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
- 将KL约束下的奖励最大化解析解融入梯度权重，确保最优策略对齐。
- 理论保证唯一最优解，且在多种采样分布下表现稳定。
- 适合需要高可靠性的大模型对齐任务，如对话系统优化。
后训练在优化和对齐大语言模型以满足特定任务与人类偏好方面起着关键作用。尽管近期的后训练技术（如分组相对策略优化，GRPO）通过增加采样并采用相对奖励评分实现了优异性能，但这些方法常因训练不稳定性而限制实际应用。为此，我们提出分组方差策略优化（GVPO）。GVPO将KL约束下奖励最大化的解析解直接融入梯度权重，确保与最优策略一致。该方法具有直观物理意义：其梯度反映隐式奖励中心距离与实际奖励中心距离之间的均方误差。GVPO具备两大优势：(1) 保证唯一最优解，即精确匹配KL约束下的奖励最大化目标；(2) 支持灵活采样分布，避免了在线策略与重要性采样带来的局限。通过融合理论保证与实际适应性，GVPO建立了一种可靠且通用的大语言模型后训练新范式。
原文摘要 · Abstract (English)
Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。