用策略自身反馈自监督改进奖励模型，解决标注成本高问题。
The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
- 用价值函数为策略输出打分，生成自监督信号
- 在六个基准上均超越基线，三类强化学习算法稳定提升
- 适合需要低成本强化学习对齐的模型训练场景
构建强效语言模型对齐的奖励模型（RM）受限于人工标注或裁判模型获取多样化、可靠偏好数据的成本与难度。随着策略演化，静态RM训练更显不足。为此，我们提出SAVE（基于价值锚定的在线反馈自监督奖励模型优化），通过价值函数对在线策略输出进行评分，作为奖励模型训练的自监督信号。该方法以提示特定的价值头作为动态锚点，将奖励评分转化为监督信号，计算奖励优势并过滤模糊样本，通过对比目标更新奖励模型。在六个不同基准上的严格实证评估验证了其有效性：在所有数据集上表现均优于基线，且在三种强化学习算法（GRPO、RLOO、GSPO）和不同策略主干网络下保持一致提升。
原文摘要 · Abstract (English)
Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。