让奖励模型像人一样分两步思考,更准地发现隐藏错误。
Think Twice: Branch-and-Rethink Reasoning Reward Model
- 分两轮评估:先选关键维度,再针对性重审证据。
- 在三个基准上超越现有方法,对细微错误更敏感。
- 适合需要高可靠性推理的场景,如安全审核与复杂决策。
大型语言模型越来越多依赖思维模型,在推理时外化中间步骤并分配额外计算资源。研究发现,有意识地进行第二次审视(think-twice)能显著提升推理质量。然而,多数奖励模型仍采用一次性评分方式,将多种质量维度压缩为单一标量,导致判断力分散、分析浅显。本文提出分支-重思(Branch-and-Rethink, BR-RM)奖励模型,将 think-twice 原则引入奖励建模:第一轮自适应分支,选出少数关键维度(如事实性、安全性),并提出简明的证据验证假设;第二轮基于分支条件重思,针对性重新阅读并检验这些假设。我们采用类似 GRPO 的强化学习训练结构化两轮轨迹,使用二值奖励与严格格式检查,兼容标准 RLHF 流程。通过将一次性评分转为聚焦的二次推理,BR-RM 减少了判断力分散,提升了对微小但关键错误的敏感度,同时保持实用性和可扩展性。实验表明,该模型在三个跨领域挑战性奖励建模基准上达到当前最优性能。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly rely on thinking models that externalize intermediate steps and allocate extra test-time compute, with think-twice strategies showing that a deliberate second pass can elicit stronger reasoning. In contrast, most reward models (RMs) still compress many quality dimensions into a single scalar in one shot, a design that induces judgment diffusion: attention spreads across evaluation criteria, yielding diluted focus and shallow analysis. We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling. Turn 1 performs adaptive branching, selecting a small set of instance-critical dimensions (such as factuality and safety) and sketching concise, evidence-seeking hypotheses. Turn 2 executes branch-conditioned rethinking, a targeted reread that tests those hypotheses and scrutinizes only what matters most. We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks, making the approach compatible with standard RLHF pipelines. By converting all-at-once scoring into focused, second-look reasoning, BR-RM reduces judgment diffusion and improves sensitivity to subtle yet consequential errors while remaining practical and scalable. Experimental results demonstrate that our model achieves state-of-the-art performance on three challenging reward modeling benchmarks across diverse domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。