用分位数回归建模奖励分布，更精准捕捉人类偏好多样性。
Quantile Regression for Distributional Reward Models in RLHF
- 采用分位数回归学习奖励的完整分布，而非单一数值。
- 在RewardBench上优于传统点估计模型，降低极端负面输出。
- 适合需要风险感知的强化学习场景，如安全可控生成。
基于人类反馈的强化学习（RLHF）已成为对齐大语言模型与人类偏好的关键方法，依赖于奖励模型。然而，传统奖励模型通常仅输出点估计，难以反映人类价值观的多样性和复杂性。本文提出分位数奖励模型（QRMs），通过分位数回归学习奖励的完整分布（可能为多模态），提供更丰富、细腻的偏好表征。该方法能更好捕捉人类偏好的多样性，缓解标签噪声，并将冲突偏好建模为分布中的不同模式。实验表明，QRM在RewardBench上优于同类点估计模型。此外，分布估计提供的额外信息可用于下游任务，如风险感知强化学习，使大模型生成更少极端负面回应。代码与模型已开源。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) has become a key method for aligning large language models (LLMs) with human preferences through the use of reward models. However, traditional reward models typically generate point estimates, which oversimplify the diversity and complexity of human values and preferences. In this paper, we introduce Quantile Reward Models (QRMs), a novel approach to reward modeling that learns a distribution over rewards instead of a single scalar value. Our method uses quantile regression to estimate a full, potentially multimodal distribution over preferences, providing a more powerful and nuanced representation of preferences. This distributional approach can better capture the diversity of human values, addresses label noise, and accommodates conflicting preferences by modeling them as distinct modes in the distribution. Our experimental results show that QRM outperforms comparable traditional point-estimate models on RewardBench. Furthermore, we demonstrate that the additional information provided by the distributional estimates can be utilized in downstream applications, such as risk-aware reinforcement learning, resulting in LLM policies that generate fewer extremely negative responses. Our code and model are released at https://github.com/Nicolinho/QRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。