让大模型推理时区分关键步骤和废话，提升学习效率。
Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization
- 用反事实方法评估每个词的重要性，动态调整梯度更新
- 在GSM8K上比均匀赋权快收敛，准确率更高
- 无需额外模型，自动从自身概率变化中提取重要性
语言模型推理中的策略梯度方法（如GRPO和DAPO）对所有生成的词统一赋予权重，导致像“让我想想”这样的填充语与“23 + 45 = 68”这类关键计算获得相同梯度更新。本文提出反事实重要性加权：遮蔽推理片段，测量答案概率下降程度，并据此在策略梯度更新中放大关键词的权重。该方法无需辅助模型或人工标注，重要性直接由策略模型自身的概率变化估计。在涵盖Qwen和Llama系列三个模型的GSM8K数据集上，实验显示该方法持续优于均匀基线，且收敛更快至相当准确率。反转重要性信号会损害性能，验证了所捕获的是真实的因果结构而非噪声。分析表明，该方法能正确识别计算步骤优于支架文本。研究认为反事实重要性加权是未来研究的基础，而非完整解决方案。
原文摘要 · Abstract (English)
Policy gradient methods for language model reasoning, such as GRPO and DAPO, assign uniform credit to all generated tokens - the filler phrase "Let me think" receives the same gradient update as the critical calculation "23 + 45 = 68." We propose counterfactual importance weighting: mask reasoning spans, measure the drop in answer probability, and upweight tokens accordingly during policy gradient updates. Our method requires no auxiliary models or external annotation, instead importance is estimated directly from the policy model's own probability shifts. Experiments on GSM8K across three models spanning the Qwen and Llama families demonstrate consistent improvements over uniform baselines and faster convergence to equivalent accuracy. Inverting the importance signal hurts performance, confirming we capture genuine causal structure rather than noise. Analysis shows the method correctly prioritizes calculation steps over scaffolding text. We view these findings as establishing counterfactual importance weighting as a foundation for further research rather than a complete solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。