用几何平均提升大模型推理稳定性，解决奖励异常导致的训练不稳问题。
Geometric-Mean Policy Optimization
- 以几何平均替代算术平均优化令牌奖励，降低异常值影响
- 在多个数学推理任务上，70亿参数模型提升4.1%通过率
- 可直接替换GRPO，无需修改结构，适合追求稳定性的强化学习应用
组相对策略优化（GRPO）通过优化令牌级奖励的算术平均，显著提升了大语言模型的推理能力。然而，当面对重要性加权奖励存在异常值时，GRPO会出现策略更新不稳定的问题，表现为训练中重要性采样比率极端。本文提出几何平均策略优化（GMPO），通过最大化令牌级奖励的几何平均来抑制奖励异常值。与算术平均不同，几何平均对异常值不敏感，能保持更重要的采样比率在更稳定的范围内。GMPO仅需将GRPO中的算术平均替换为几何平均，即可实现即插即用。理论分析表明，两者均为加权策略梯度形式，但GMPO具有更稳定的权重。在多个数学推理基准测试中，GMPO-7B模型的平均Pass@1指标比GRPO最高提升4.1%，超越多项现有先进方法。代码已公开于https://github.com/callsys/GMPO。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has significantly enhanced the reasoning capability of large language models by optimizing the arithmetic mean of token-level rewards. Unfortunately, GRPO is observed to suffer from unstable policy updates when facing tokens with outlier importance-weighted rewards, which manifest as extreme importance sampling ratios during training. In this study, we propose Geometric-Mean Policy Optimization (GMPO), with the aim to improve the stability of GRPO through suppressing token reward outliers. Instead of optimizing the arithmetic mean, GMPO maximizes the geometric mean of token-level rewards, which is inherently less sensitive to outliers and maintains a more stable range of importance sampling ratio. GMPO is plug-and-play-simply replacing GRPO's arithmetic mean with the geometric mean of token-level rewards, as the latter is inherently less sensitive to outliers. GMPO is theoretically plausible-analysis reveals that both GMPO and GRPO are weighted forms of the policy gradient while the former enjoys more stable weights, which consequently benefits policy optimization and performance. Experiments on multiple mathematical reasoning benchmarks show that GMPO-7B improves the average Pass@1 of GRPO by up to 4.1%, outperforming many state-of-the-art approaches. Code is available at https://github.com/callsys/GMPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。