arXiv:2606.03980cs.LGcs.CL2026-06被引 3

用可复用的技能统一多种评价标准,让大模型奖励模型更智能。

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

论文配图:Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
图 1 · 摘自论文原文
  • 将奖励计算视为可复用的智能体任务,动态调用不同证据。
  • 在多个基准上优于传统判别器,提升最佳选择与强化学习效果。
  • 适合需要多类型评价融合的智能体训练场景。

奖励模型(RMs)为大语言模型后训练提供关键反馈信号,尤其在强化微调(RFT)和强化学习(RL)流程中。然而当前奖励评估依赖规则验证器、真实参考、步骤清单和复杂评分标准等异构标准,缺乏统一整合机制。为此,我们提出技能奖励模型(Skill-RM),将奖励建模重构为可复用的“奖励-评估技能”执行过程。通过将奖励计算视为结构化智能体任务,Skill-RM为异构资源提供一致接口,动态选择并聚合适配输入需求的证据。该方法使奖励模型超越静态评估,实现跨多样化任务的一致性与透明性。在多个奖励基准及下游应用(包括最佳N选一和强化学习)中的广泛实验表明,Skill-RM始终优于传统裁判基线。结果表明,Skill-RM不仅提供统一的奖励建模方案,还通过策略性、动态化的证据调度实现更优性能。代码已开源:https://github.com/Qwen-Applications/Skill-RM。

原文摘要 · Abstract (English)

Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics, where a unified mechanism to integrate all types of evidence remains unexplored. To this end, we propose Skill Reward Model (Skill-RM), a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. By treating reward computation as a structured agentic task, Skill-RM provides a consistent interface to orchestrate heterogeneous resources, dynamically selecting and aggregating evidence tailored to the specific requirements of each input. This approach enables the reward model to move beyond static evaluation, ensuring consistency and transparency across diverse tasks. Extensive experiments on reward benchmarks and downstream applications, including best-of-N selection and reinforcement learning, demonstrate that Skill-RM consistently outperforms traditional judge baselines. Our findings suggest that Skill-RM not only provides a unified solution for reward modeling but also achieves superior performance through the strategic and dynamic orchestration of evidence. The code is at https://github.com/Qwen-Applications/Skill-RM.

奖励模型智能体统一评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。