小模型可当评分裁判，省钱又高效。
Small Language Models as Judges for Rubric-Based Reinforcement Learning
- 用小模型提取评分标准判断，三种方法比拼
- 17亿参数小模型在两项数据集上表现最佳
- 训练效率提升10倍，适合资源有限的研究者
基于评分标准的强化学习拓展了强化学习在无明确答案任务中的应用，通过针对具体实例的标准打分实现。但该方法奖励计算成本高：训练需反复调用评分标准评判，通常依赖大模型（70亿参数及以上）生成式裁判或专有接口。本文研究小语言模型能否作为高效且可靠的评分裁判。为使问题可衡量，构建了两个点对点评分标准评估数据集：PointRubric 和 RaR-Science-Static，包含实例级标准与逐项满足标签。比较三种从小型模型中提取标准级判断的方法：生成式结论、是/否对数概率差、探针裁判。在两个数据集上，Qwen3-1.7B 探针裁判在标准级一致性上表现最强，优于生成式和对数概率裁判。作为 GRPO 奖励模型，其将政策在 RaR-Science 评分上的得分从 0.232 提升至 0.643，优于 80亿参数生成式裁判基线（0.594），且基线所需评分时间多出 10.7 倍。跨任务与领域迁移实验进一步表明，探针裁判能保持标准级奖励结构的一致性。
原文摘要 · Abstract (English)
Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7$\times$ more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。