用快慢思维融合标量与生成式奖励模型，提升效率与性能。
Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models
- 快慢双思架构：先快速预测得分，复杂时启动链式推理判断。
- 性能提升1.2%，令牌消耗降低20.8%。
- 适合追求高效高精度对齐的RLHF应用。
奖励模型（RMs）在通过人类反馈强化学习（RLHF）对齐大语言模型中至关重要。生成式奖励模型（GRMs）虽通过链式思维（CoT）推理实现更优准确率，但计算成本高昂；标量奖励模型（SRMs）虽高效，但在复杂场景下表现有限且适应性差。本文提出快-慢思维奖励模型（F/S-RM），受双重认知理论启发，训练单一模型融合两种奖励范式：以首词预测作为标量评分（快思考），在必要时启用基于链式思维的判断（慢思考）。该模型由双置信度激活机制调控慢思考触发时机。F/S-RM相比当前最优模型实现1.2%相对性能提升，同时减少20.8%的令牌消耗。代码与数据将公开。
原文摘要 · Abstract (English)
Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: first-token prediction as a scalar score (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. F/S-RM achieves a 1.2% relative performance improvement over state-of-the-art models while reducing token consumption by 20.8%. Code and data will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。