arXiv:2601.07408cs.CLcs.LG2026-01ACL被引 17

提出细粒度奖励重分配方法,让大模型解题时更精准识别关键步骤。

Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning

  • 基于答案敏感性重新分配奖励,区分每一步推理的重要程度。
  • 两种实现方式:扰动法(高精度)和梯度代理法(低开销),均显著提升性能。
  • 适合想优化大模型数学推理能力的研究者与工程师使用。

群体相对策略优化(GRPO)是一种有前景的无评价值函数强化学习范式,适用于推理任务。但标准GRPO采用粗粒度信用分配机制,将群体奖励均匀传递给序列中每个标记,忽略了单个推理步骤的差异性贡献。为此,本文提出基于结果的奖励重分配(OAR),一种细粒度信用分配机制,依据每个标记对最终答案的影响程度重新分配优势值。OAR通过两种互补策略实现:(1) OAR-P,通过反事实标记扰动估计结果敏感性,提供高保真归因信号;(2) OAR-G,使用输入梯度敏感性代理,在一次反向传播中近似影响信号。这些重要性信号与保守的双层优势重分配方案结合,抑制低影响标记,增强关键步骤,同时保持整体优势质量。在多个数学推理基准上的实验表明,尽管OAR-P达到性能上限,但OAR-G以可忽略的计算开销实现相当提升,显著优于强基线GRPO,推动了无评价值大模型推理的边界。

原文摘要 · Abstract (English)

Group Relative Policy Optimization (GRPO) has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks. However, standard GRPO employs a coarse-grained credit assignment mechanism that propagates group-level rewards uniformly to to every token in a sequence, neglecting the varying contribution of individual reasoning steps. We address this limitation by introducing Outcome-grounded Advantage Reshaping (OAR), a fine-grained credit assignment mechanism that redistributes advantages based on how much each token influences the model's final answer. We instantiate OAR via two complementary strategies: (1) OAR-P, which estimates outcome sensitivity through counterfactual token perturbations, serving as a high-fidelity attribution signal; (2) OAR-G, which uses an input-gradient sensitivity proxy to approximate the influence signal with a single backward pass. These importance signals are integrated with a conservative Bi-Level advantage reshaping scheme that suppresses low-impact tokens and boosts pivotal ones while preserving the overall advantage mass. Empirical results on extensive mathematical reasoning benchmarks demonstrate that while OAR-P sets the performance upper bound, OAR-G achieves comparable gains with negligible computational overhead, both significantly outperforming a strong GRPO baseline, pushing the boundaries of critic-free LLM reasoning.

强化学习数学推理信用分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。