利用奖励模型隐状态提升强化学习的奖励估计精度
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
- 将奖励模型隐藏层特征建模为图结构，通过节点间相似性传播优势值
- 在三个基准上实现最高+8.27的提升，尤其在Arena-Hard上达+6.3
- 轻量级设计，可无缝集成到现有群组强化学习算法中
当前基于人类反馈的强化学习（RLHF）主要依赖训练好的奖励模型（RM）输出的标量奖励。尽管有效，但标量奖励常存在噪声且难以捕捉细微偏好差异，而RM的隐藏状态则包含更丰富的语义与偏好信息。本文提出表示感知的优势估计方法，将RM隐藏状态作为辅助信号以优化优势估计。具体地，提出图式优势估计（GraphAE），将每个采样组视为图，节点代表响应，边表示其在RM隐藏空间中的相似性。通过图传播计算优势，使每个样本能吸收邻近样本的上下文信息。GraphAE轻量且可无缝集成至现有群组基强化学习算法。我们将其应用于GRPO、GSPO和RLOO，在不同模型与基准上进行大量实验。结果表明，三组基准上均有持续提升，最大增益达+6.3（Arena-Hard-v0.1）、+8.27（AlpacaEval 2.0）和+0.22（MT-Bench）。结果证明，利用RM表示可实现更高效、更鲁棒的RLHF。
原文摘要 · Abstract (English)
Current reinforcement learning from human feedback (RLHF) methods primarily rely on scalar rewards from a trained reward model (RM). While effective, scalar rewards are often noisy and fail to capture fine-grained preference differences, whereas RM hidden states encode richer semantic and preference information. We introduce the representation-aware advantage estimation, which leverages RM hidden states and models them as auxiliary signals for better advantage estimation. Specifically, we propose the Graph-based Advantage Estimation (GraphAE), treat each sampled group as a graph, where nodes correspond to responses and edges capture their similarity in the RM hidden space. Then advantages are computed via graph propagation, enabling each sample to incorporate contextual information from its neighbors. GraphAE is lightweight and can be seamlessly integrated into existing group-based RL algorithms. We apply GraphAE to GRPO, GSPO and RLOO, and conduct extensive experiments on different models and benchmarks. Empirical results show consistent improvements across three benchmarks, with gains of up to + 6.3 on Arena-Hard-v0.1, + 8.27 on AlpacaEval 2.0, and + 0.22 on MT-Bench. These results demonstrate that leveraging RM representations leads to more sample efficient and robust RLHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。