让奖励模型像人一样思考后再打分，提升对齐效果。
Reward Reasoning Model
- 引入链式思维推理，用额外计算资源做复杂判断
- 在多个领域基准上表现优于现有方法，且可自适应调用算力
- 无需标注推理过程，通过强化学习自主进化推理能力
奖励模型在引导大语言模型输出符合人类期望方面起关键作用。然而，在测试阶段有效利用计算资源以提升奖励模型性能仍是开放挑战。本文提出奖励推理模型（RRMs），专门设计用于在生成最终奖励前执行严谨的推理过程。通过链式思维推理，RRMs可在奖励不明确的复杂查询中利用额外的测试时计算。为构建RRMs，我们采用强化学习框架，使模型在无需显式推理标注数据的情况下自主发展推理能力。实验结果表明，RRMs在跨多个领域的奖励建模基准上均取得更优表现。值得注意的是，我们证明了RRMs能自适应地利用测试时计算进一步提升奖励准确性。预训练的奖励推理模型已开源：https://huggingface.co/Reward-Reasoning。
原文摘要 · Abstract (English)
Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。