用二值化反馈融合人类偏好与可验证奖励，提升大模型对回答质量的精准理解。
RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
- 从自然语言反馈中提取可二值判断的原则，用于训练奖励模型
- 在RM-Bench和JudgeBench上分别达86.2%和81.4%，位列榜首
- 支持推理时自定义关注原则，且开源方案成本低于5%
基于人类反馈的强化学习（RLHF）与基于可验证奖励的强化学习（RLVR）是大模型后训练的主要范式，各有优劣。RLHF因依赖缺乏明确标准的人类判断，存在可解释性差和奖励黑客问题；而RLVR受限于仅关注正确性验证，适用范围窄。本文提出二值灵活反馈强化学习（RLBFF），将人类偏好灵活性与规则验证精确性结合，使奖励模型能捕捉超越单纯正确性的响应质量细节。RLBFF从自然语言反馈中提取可二值回答的原则（如‘信息准确：是’或‘代码可读性：否’），并将其作为蕴含任务训练奖励模型（响应满足或不满足任意原则）。实验表明，该方法在数据量相当条件下优于布拉德利-特里模型，在RM-Bench（86.2%）和JudgeBench（81.4%，截至2025年9月24日榜首）表现顶尖。用户可在推理时指定关注原则，实现动态调整。此外，我们提供完整开源方案（含数据），使用RLBFF对Qwen3-32B进行对齐，其性能在MT-Bench、WildBench和Arena Hard v2上达到或超过o3-mini和DeepSeek R1，推理成本低于5%。
原文摘要 · Abstract (English)
Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) are the main RL paradigms used in LLM post-training, each offering distinct advantages. However, RLHF struggles with interpretability and reward hacking because it relies on human judgments that usually lack explicit criteria, whereas RLVR is limited in scope by its focus on correctness-based verifiers. We propose Reinforcement Learning with Binary Flexible Feedback (RLBFF), which combines the versatility of human-driven preferences with the precision of rule-based verification, enabling reward models to capture nuanced aspects of response quality beyond mere correctness. RLBFF extracts principles that can be answered in a binary fashion (e.g. accuracy of information: yes, or code readability: no) from natural language feedback. Such principles can then be used to ground Reward Model training as an entailment task (response satisfies or does not satisfy an arbitrary principle). We show that Reward Models trained in this manner can outperform Bradley-Terry models when matched for data and achieve top performance on RM-Bench (86.2%) and JudgeBench (81.4%, #1 on leaderboard as of September 24, 2025). Additionally, users can specify principles of interest at inference time to customize the focus of our reward models, in contrast to Bradley-Terry models. Finally, we present a fully open source recipe (including data) to align Qwen3-32B using RLBFF and our Reward Model, to match or exceed the performance of o3-mini and DeepSeek R1 on general alignment benchmarks of MT-Bench, WildBench, and Arena Hard v2 (at <5% of the inference cost). Models: https://huggingface.co/collections/nvidia/reward-models-10-2025
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。