提出GTPO方法，让大模型对齐训练更稳定、效果更好。
GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
- 通过跳过负梯度更新，避免关键词被错误惩罚。
- 用熵阈值过滤低质量生成，防止模型决策崩溃。
- 无需参考模型和KL正则，适合高效训练大模型。
组相对策略优化（GRPO）是一种有前景的大语言模型对齐方法，但常受训练不稳定性与收敛不佳的限制。本文分析发现两个核心问题：(i) token级惩罚导致跨响应共享的有价值token收到矛盾反馈，引发冲突梯度更新，降低其出现概率；(ii) 策略坍缩现象中，负奖励的完成句会惩罚高置信度输出，使模型倾向低概率词，破坏训练过程。为此，我们提出GTPO（组相对轨迹策略优化），通过跳过负更新、强化正向更新来保护关键token的梯度一致性，并引入可证明熵阈值过滤掉低熵完成句，防止策略坍缩。与GRPO不同，GTPO无需KL散度正则化，也无需参考模型进行训练，同时在GSM8K、MATH、AIME 2024、AIME 2025和AMC 2023等多个数据集上验证了更高的训练稳定性和性能提升。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is a promising policy-based approach for Large Language Model alignment, yet its performance is often limited by training instability and suboptimal convergence. In this paper, we identify and analyze two main GRPO issues: (i) the token-level penalization, where valuable tokens shared across different responses receive contradictory feedback signals, leading to conflicting gradient updates that can reduce their likelihood; and (ii) the policy collapse, where negatively rewarded completions may penalize confident responses and shift model decisions toward unlikely tokens, destabilizing training process. To address these issues we introduce GTPO (Group-relative Trajectory-based Policy Optimization), which prevents conflicting gradients on valuable tokens by skipping negative updates while amplifying positive ones and filters out completions whose entropy exceeds a provable threshold, to prevent policy collapse. Unlike GRPO, GTPO does not rely on KL-divergence regularization, eliminating the need for a reference model during training, while still ensuring greater training stability and improved performance, as validated through multiple experiments on GSM8K, MATH, AIME 2024, AIME 2025 and AMC 2023.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。