把评估指标当奖励模型用，能避免虚假相关和奖励黑客问题
Reward Models are Metrics in a Trench Coat
- 将评估指标与奖励模型融合，共享优化思路
- 实验证明某些指标在特定任务上优于传统奖励模型
- 适合研究大模型训练信号、评估机制的学者参考
大语言模型后训练中强化学习的兴起引发了对奖励模型的广泛关注。奖励模型用于评估生成输出的质量以提供训练信号，这一任务也由评估指标完成。我们发现这两个领域大多独立发展，导致术语重复和共性问题频发。常见挑战包括对虚假相关性的敏感、下游奖励黑客风险、数据质量提升方法及元评估策略。本文主张两领域加强协作以克服上述问题。我们展示某些指标在特定任务上优于奖励模型，并系统梳理了两个方向的研究进展。基于此，我们指出多个可协同改进的领域，如偏好获取方法、规避虚假相关与奖励黑客、以及校准感知的元评估。
原文摘要 · Abstract (English)
The emergence of reinforcement learning in post-training of large language models has sparked significant interest in reward models. Reward models assess the quality of sampled model outputs to generate training signals. This task is also performed by evaluation metrics that monitor the performance of an AI model. We find that the two research areas are mostly separate, leading to redundant terminology and repeated pitfalls. Common challenges include susceptibility to spurious correlations, impact on downstream reward hacking, methods to improve data quality, and approaches to meta-evaluation. Our position paper argues that a closer collaboration between the fields can help overcome these issues. To that end, we show how metrics outperform reward models on specific tasks and provide an extensive survey of the two areas. Grounded in this survey, we point to multiple research topics in which closer alignment can improve reward models and metrics in areas such as preference elicitation methods, avoidance of spurious correlations and reward hacking, and calibration-aware meta-evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。