arXiv:2608.16072cs.LGcs.AI2026-08

针对多奖励强化学习中优化效率低的问题,提出动态调整奖励权重的方法。

Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

论文配图:Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
图 1 · 摘自论文原文
  • 按目标独立标准化并根据饱和度自适应降低已满足目标的权重
  • 在15个测试中12次提升更难任务准确率,最高增5%
  • 可反转梯度方向,适合复杂多目标优化场景

基于组内相对优势的强化学习已成为语言模型推理后训练的标准方法。然而,在优化多个奖励目标时,现有方法通常在组间标准化前对奖励向量进行固定加权求和。我们指出该设计导致两个根本问题:具有不同奖励分布的轨迹可能获得相同优势,且所有目标始终以固定相对权重优化,无论其是否已趋于饱和。结果是训练持续将梯度预算分配给已解决的目标,而非聚焦仍有提升空间的项。为此,我们提出饱和感知优势重加权方法(SA-MRPO),对每个奖励目标独立标准化,并根据批次级目标饱和度估计动态降低其贡献权重。该方法将优化精力重新分配至未充分优化的目标,同时在已满足的目标上保持性能。进一步发现,饱和感知重加权可改变更新方向,而不仅缩放幅度。在双目标与三目标数学推理任务中,相较于GDPO,SA-MRPO在15个基准测试中有12次提升更难正确性目标,最高达5%(AIME24)。在自适应推理任务中,五个基准平均提升3.8%,最高达9.2%(AMC23);在编码任务中,通过率最高提升2.3%,且所有设置下均维持简单目标接近已满足水平。

原文摘要 · Abstract (English)

Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce \textbf{Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization} (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to $5\%$ on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by $3.8\%$ on average and up to $9.2 \%$ on AMC23, and on coding benchmarks it improves pass rate by up to $2.3\%$, while in all settings maintaining the easier objectives near their already satisfied levels.

强化学习多目标优化奖励设计语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。