提出结构化奖励模型,让机器评分更透明、高效且可扩展。
Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
- 用模块化分支生成关键特征,实现细粒度维度评估
- 在真实工业场景中表现优于传统评分模型,人类偏好对齐更好
- 适合需要精准诊断的推荐、搜索等单领域任务
奖励模型(RMs)是评估和引导语言模型输出的核心组件。传统标量奖励模型在推理时难以融入上下文信息,导致评估不完整。生成式奖励模型(GRMs)通过生成中间推理步骤试图解决此问题,但其不可控的黑箱特性及串行解码带来的效率低下,限制了工业应用。在搜索、推荐等单领域任务中,需按特定维度评估输出,诊断‘差例’需结构化反馈以定位并优化具体问题。本文提出结构化奖励模型(SRM),采用模块化设计,引入侧分支模型作为辅助特征生成器,通过细粒度维度实现可解释、高效的评估,支持针对性诊断与优化。实验表明,SRM在鲁棒性和人类偏好对齐方面均优于标量RM和GRM,其模块化架构也便于实际场景下的高效优化,为工业级奖励建模提供可行方案。
原文摘要 · Abstract (English)
Reward Models (RMs) are key components for evaluating and guiding language model outputs. However, traditional scalar RMs often struggle with incorporating contextual and background information during inference, leading to incomplete evaluations. Generative RMs (GRMs) attempt to address these limitations by generating intermediate reasoning steps. Yet, their uncontrolled black-box nature and inefficiency due to sequential decoding hinder their industrial deployment. Industrial scenarios, such as search and recommendation systems, often involve single-domain tasks requiring evaluation along specific dimensions. In such contexts, diagnosing "bad cases" necessitates structured feedback to identify and optimize dimension-specific issues. In this paper, we propose the Structural Reward Model (SRM), a modular and interpretable framework integrating side-branch models as auxiliary feature generators. By introducing fine-grained dimensions, SRMs enable interpretable and efficient evaluation, facilitating targeted diagnostics and optimization. This structured approach ensures adaptability and scalability for industrial applications. Through comprehensive experiments, we demonstrate that SRMs outperform scalar RMs and GRMs in robustness and alignment with human preferences. The modular design further supports efficient optimization for practical scenarios, allowing SRM to provide a practical reward modeling solution for industry.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。