用强化学习训练推理型大模型当裁判，比传统方法更擅长复杂判断。
JudgeLRM: Large Reasoning Models as a Judge
- 通过强化学习和任务导向奖励，激活模型的深层推理能力。
- 小模型（3B/4B）超越GPT-4，大模型（7B以上）比DeepSeek-R1高2%以上F1。
- 特别适合需要逻辑验证、错误识别的高阶评估任务。
大语言模型正被广泛用作评估工具，提供可扩展的替代人工标注方案。然而，现有监督微调方法在需要复杂推理的领域表现有限。评估本质是推理密集型任务：不仅需表面打分，还需验证证据、识别错误并给出理由。分析显示，监督微调性能提升与推理样本占比呈负相关，揭示其在该类场景下的局限性。为此，我们提出JudgeLRM，一种以判别为导向的大模型家族，采用强化学习与判官级、结果驱动的奖励机制，激发推理能力。JudgeLRM在同规模下持续优于监督微调基线及其他强化学习与微调变体，甚至超越当前最优推理模型：其中，JudgeLRM-3B/4B超过GPT-4，JudgeLRM-7B/8B/14B在F1得分上比DeepSeek-R1高出2%以上，尤其在高推理强度任务中优势显著。研究证明，强化学习对实现推理对齐的评估模型具有关键价值。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。