解决大模型微调中梯度冲突问题，提升优化稳定性。
GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
- 将梯度视为随机变量，用贝叶斯方法建模不确定性
- 通过狄利克雷分布估计梯度可靠性，动态调整贡献权重
- 在多个基准上验证有效，尤其在梯度冲突时表现更优
组相对策略优化（GRPO）已成为大语言模型推理能力后训练的常用方法。在GRPO中，同一小批量内不同查询产生的组梯度被直接平均以形成策略更新。然而，这些组梯度可能指向冲突方向。我们实证分析发现，梯度冲突常与低效的策略更新相关，亟需在冲突下获得可靠的聚合更新方向。标准GRPO将实现的组梯度视为确定性贡献，未考虑其可靠性差异。为此，我们提出梯度不确定性感知策略优化（GUPO），在贝叶斯框架下将每个组梯度建模为随机变量，并估计其概率分布。GUPO基于狄利克雷分布推导梯度不确定性，并据此校准各组梯度在聚合中的贡献。大量实验在多个基准上验证了其有效性。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。