用评分标准设计奖励模型，防止大模型为了高分乱答。
Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
- 用评分标准提取高质量回复特征，避开低质量高分陷阱。
- 在高分尾部区域提升区分度，减少模型作弊行为。
- 适合想提升大模型输出质量的研究者和工程师。
强化学习微调常因奖励过优化而出现模型作弊现象：为追求高分生成低质量内容。理论分析表明，问题根源在于高分尾部的奖励误设——无法可靠区分‘优秀’与‘良好’回复。为此，我们聚焦高分区域，提出基于评分标准的奖励建模方法。通过利用外部强模型或重写样本生成稀有的高分示例，结合评分标准设计，使奖励信号对这些外源样本的噪声不敏感。我们设计了一套流程，强调对多样且优秀的回复进行精细区分。实验表明，该方法显著缓解奖励过优化问题，有效提升大模型后训练性能。
原文摘要 · Abstract (English)
Reinforcement fine-tuning (RFT) often suffers from reward over-optimization, where a policy model hacks the reward signals to achieve high scores while producing low-quality outputs. Our theoretical analysis shows that the key lies in reward misspecification at the high-reward tail: the inability to reliably distinguish Excellent responses from merely Great ones. This motivate us to focus on the high-reward region. However, such tail examples are scarce under the base LLM. While off-policy exemplars (e.g. from stronger models or rewrites) are easier to obtain, naively training on them yields a misspecified reward for the policy we aim to align. To address this, we study rubric-based rewards. By design, rubrics can leverage off-policy examples while remaining insensitive to their artifacts. To elicit rubrics that capture the high-reward tail, we highlight the importance of distinguishing among great and diverse responses, and introduce a workflow to implement this idea. We empirically demonstrate that rubric-based rewards substantially mitigate reward over-optimization and deliver effective LLM post-training improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。