arXiv:2605.20164cs.AI2026-05被引 4

动态调整评分权重,让强化学习更高效地学好复杂任务。

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

论文配图:Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
图 1 · 摘自论文原文
  • 根据策略表现动态调整评分标准权重,避免无效信号干扰
  • 在30次对比中胜出24次,完成率提升且训练速度加快2.5至4倍
  • 适合需要多维度评估的生成模型优化场景

基于可验证奖励的强化学习在自动校验正确性时非常有效。但许多重要行为需同时满足多个定性标准。传统评分式奖励通过加权聚合各标准为单一奖励,却将人类赋予的重要性与当前优化信号有效性混为一谈。我们发现,在评分型强化学习中,许多标准已饱和或不可达,而真正能区分输出的标准未必是权重最高的。为此提出POW3R框架:在保持人类权重和类别平衡的前提下,根据回滚结果的对比动态调整各标准权重,使GRPO奖励更具信息量。在两个数据集、三种基础策略上的实验表明,POW3R在30组比较中胜出24组,显著提升平均评分与严格完成率(所有标准均满足的比例),且达到相同性能仅需2.5至4倍少的训练步数。评分奖励应区分最终答案的关键项与当前可指导策略的学习项。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins $24$ of $30$ base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in $2.5$--$4\times$ fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

强化学习评分机制生成模型奖励设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。