arXiv:2605.17291cs.LG2026-05被引 7

给大模型推理每一步打分,让奖励更精准,避免错误误导。

Step-wise Rubric Rewards for LLM Reasoning

论文配图:Step-wise Rubric Rewards for LLM Reasoning
图 1 · 摘自论文原文
  • 按步骤分配评分标准,让每步推理都有独立奖励信号。
  • 在六项数学题上准确率提升2.75至3.57点,错误自修正减少一半。
  • 适合需要严谨推理的AI研究者和算法开发者。

强化学习结合可验证奖励(RLVR)常用于提升大语言模型的推理能力,但仅根据最终答案正确性给予奖励,缺乏对中间步骤的监督。基于评分表的方法如RaR虽引入细粒度标准,但将评分汇总为单一标量,导致三个问题:多标准结构丢失、正确与错误步骤受到相同惩罚、可通过无限自我修正进行奖励作弊。在1000道题目中,我们发现18.2%的正确答案包含错误步骤却获正向奖励,49.9%的错误答案中存在正确步骤却被惩罚。为此提出分步评分奖励(SRaR),通过大模型裁判将每个评分项关联到具体推理步骤,对每步评分进行跨轨迹归一化,仅使质量变化的步骤产生学习信号,并采用解耦优势估计器融合步骤奖励与结果奖励,保持结果基线稳定。我们还构建了一个包含16,000道题目的评分表数据集,通过对比蒸馏从强模型的正确与错误推理路径中提取评分项。在六个数学推理基准测试中,SRaR在Qwen3-8B上比RaR平均提升3.57点准确率,在Qwen3-32B上提升2.75点,AIME 2025的忠实推理率从34.5%升至46.7%,自我修正循环次数由48.1%降至26.5%。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.

大模型推理强化学习评分机制数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。