改进强化学习中的分组奖励机制，让模型更专注难问题。
SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation
- 用温度缩放的softmax替代z-score归一化，避免简单任务权重过高。
- 在二元奖励下实现51.8%准确率，在诗歌生成上提升至68.0%。
- 适合需要精准分配训练信号的复杂推理任务，如数学与创作。
基于分组的强化学习目标（如GRPO）在处理不同难度提示时学习信号分配不佳：在二元奖励下，组归一化会导致对简单提示的权重发散。本文提出Softmax Advantage Group Estimation（SoftmaxGRPO），一种可直接替换的方案，将z-score归一化的组优势替换为温度缩放的softmax优势，确保权重始终有界，不受提示难度影响。针对二元奖励，推导出精确的有限组群体目标，并识别出MaxRL为其低温极限。对于有界标量奖励，证明大组更新恰好优化了对数矩生成函数目标；而在无额外奖励分布假设下，通用的有限组标量目标不存在。实验表明，SoftmaxGRPO能有效将梯度预算从近似解决的提示中重新分配，并在相同奖励条件下持续优于GRPO。在DeepMath数据集上达到51.8%的准确率，使用轻量级文本相似性奖励将1.5B指令微调模型在Poetry任务上的表现从35.0%提升至68.0%。
原文摘要 · Abstract (English)
Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using only lightweight text-similarity rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。