让大模型推理更精准，通过重加权关键步骤的梯度信号。
GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards
- 基于梯度激活敏感性，动态重加权每个词元的优化强度。
- 在五种模型上平均提升准确率3.60%、Pass@3指标3.05%。
- 无需复杂奖励模型，适合希望提升逻辑推理能力的研究者。
以可验证奖励为基础的强化学习（如GRPO）已成为提升大语言模型数学推理能力的常用方法。然而，现有方法通常将序列级优势广播至所有词元，或依赖代价高昂的过程奖励模型进行步骤级监督。统一的优势分配假设所有词元对最终奖励贡献相同，这稀释了梯度信号，导致错误推理步骤与冗余词汇被同等更新。为此，我们提出梯度重加权优势（GRAIL），一种内在的词元级优势重加权方法。GRAIL利用梯度激活显著性，为对最终答案更局部敏感的词元赋予更高权重。在Qwen3、R1-distilled和OctoThinker系列共五种模型上的评估显示，GRAIL始终优于GRPO。GRAIL在准确率上平均提升3.60%，在Pass@3指标上提升3.05%，证明无需过程级监督即可实现细粒度推理对齐。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical reasoning in Large Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained reasoning alignment can be achieved without process-level supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。