让大模型自己生成批评意见，提升奖励模型的准确性
Self-Generated Critiques Boost Reward Modeling for Language Models
- 让模型自动生成自然语言批评，同时预测评分
- 相比传统方法，评分准确率提升3.7%~7.3%
- 适合需要高质量反馈的RLHF场景
奖励建模在大语言模型对齐人类偏好中至关重要，尤其在基于人类反馈的强化学习（RLHF）中。然而，现有奖励模型多输出标量分数，难以自然地融入语言形式的批评。我们提出假设：同时预测批评和标量奖励可增强建模能力。为此，我们设计Critic-RM框架，通过两阶段流程——生成并过滤高质量批评，再联合微调奖励预测与批评生成——实现无需额外监督的改进。实验表明，Critic-RM在多个基准上相比标准奖励模型和LLM裁判，奖励建模准确率提升3.7%~7.3%，展现出强性能与数据效率。进一步研究表明，生成批评能有效纠正错误推理步骤，使推理准确率提升2.5%~3.2%。
原文摘要 · Abstract (English)
Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。