用结构化评分标准让AI模型学会分步推理，提升泛化能力。
Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
- 将奖励拆解为可验证的多维度评分项，由大模型评委打分
- 在未见任务上达到71.7%的标准化奖励得分
- 适合需要可靠推理能力的科研与教育场景
我们主张将奖励分解为加权且可验证的评估标准，并使用大语言模型作为评分员进行打分，从而提供部分分式的优化信号：不再依赖二元结果或单一整体评分，而是对每个回答在多个任务相关标准上进行评分。我们形式化了“基于评分表的强化学习”（rubric-grounded RL）框架：该框架中，策略通过一个冻结的大语言模型评委生成的结构化、多准则奖励进行优化，而评委的判断基于辅助信息，这些信息是策略从未见过的。我们通过从美国科学与技术信息办公室（OSTI）提供的约10万份科技文档语料库中推导出评分表，并使用分组相对策略优化（GRPO）训练Llama-3.1-8B-Instruct模型。基于GRPO的训练使模型在保留的评分表评估中达到71.7%的标准化奖励。经过调优的策略还在四个未源自训练语料库的推理基准测试中表现优于基线模型——GSM8K、MATH、GPQA Main和GPQA Diamond。这些结果表明，基于文档的结构化奖励能提升未见任务上的评分表现，并激发超越训练语料范围的可迁移推理行为。
原文摘要 · Abstract (English)
We argue that decomposing reward into weighted, verifiable criteria and using an LLM judge to score them provides a partial-credit optimization signal: instead of a binary outcome or a single holistic score, each response is graded along multiple task-specific criteria. We formalize \emph{rubric-grounded reinforcement learning (RL)}: a framework in which the policy is optimized against a structured, multi-criterion reward produced by a frozen LLM judge that conditions on auxiliary grounding the policy never sees. We instantiate the framework by deriving rubrics from an Office of Scientific and Technical Information (OSTI)-derived corpus of roughly 100,000 scientific and technical documents and training Llama-3.1-8B-Instruct with Group Relative Policy Optimization (GRPO). With GRPO-based training, the model achieves $71.7\%$ normalized reward on held-out rubric evaluation. The GRPO-tuned policy also improves over the base model on four reasoning benchmarks not derived from the training corpus -- GSM8K, MATH, GPQA Main, and GPQA Diamond. These results provide evidence that structured, document-grounded rewards can improve held-out rubric performance and induce transferable reasoning behaviors beyond the corpus used to construct the training environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。