用梯度对齐设计密集奖励，让大模型推理更准且省时。
Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
- 基于策略梯度空间构建密集奖励，通过梯度余弦相似度衡量推理路径质量。
- 在数学竞赛题上超越GRPO等基线，8B模型提升达5.2个百分点。
- 无需额外标注，可跨领域迁移至GPQA和MMLU-Pro，适合追求高效推理的开发者。
强化学习从可验证奖励（RLVR）驱动大语言模型的思维链推理，但其二元结果奖励无法区分正确路径间的差异。现有密集奖励方法要么忽略训练语料中已有的专家解法，要么需昂贵离线标注。本文提出梯度对齐奖励（GAR），在策略自身梯度空间中运作：通过截断反向传播至输出投影层，提取每条轨迹的紧凑梯度向量，并与专家锚点梯度计算余弦相似度，生成稠密、推理感知奖励，耗时仅增加不足9%。我们证明该余弦相似度可分解为预测误差与激活模式因子的乘积，明确揭示对齐信号的构成。在Qwen3-4B和Qwen3-8B模型上，GAR在竞赛级数学基准测试中持续优于GRPO及其他基线，且无需领域特定数据即可迁移到GPQA Diamond和MMLU-Pro。代码与数据见https://github.com/LQgdwind/GAR。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。