验证性奖励未必让模型真正依赖推理,改进训练流程可让推理更可信。
Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning

- 用因果重要性和充分性衡量推理质量,发现标准强化学习训练的推理不可靠。
- 少量监督微调能提升推理的因果重要性与充分性,改善模型决策依据。
- 在结果奖励外加辅助奖励,可同时保证准确率和可验证的因果推理,适合追求可信AI的研究者。
基于可验证奖励的强化学习(RLVR)已成为语言模型后训练的标准方法,普遍认为其推理链真实反映了模型得出答案的过程。本文提出两个评估指标:因果重要性(CIR)衡量推理词元对最终答案的累积影响,充分性(SR)衡量仅凭推理能否得出明确答案。在Qwen2.5系列模型与ReasoningGym任务上的实验表明:(1)尽管RLVR提升任务准确率,但未显著改善CIR或SR,质疑推理在性能中的真实作用;(2)在RLVR前加入少量监督微调可缓解低CIR和低SR问题;(3)通过在结果奖励基础上添加辅助的CIR/SR奖励,可在不依赖SFT的情况下提升推理质量,且保持与传统RLVR相当的准确率。结果表明,标准RLVR并不能确保模型以预期方式依赖推理,但通过简单修改训练流程即可实现可验证且因果重要的推理。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought reasoning has become a standard part of language model post-training recipes. A common assumption is that the reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of Reasoning (CIR), which measures the cumulative effect of reasoning tokens on the final answer, and Sufficiency of Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) while RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of reasoning in model performance into question; (2) a small amount of SFT before RLVR can be a remedy for low CIR and SR; and (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient reasoning. These results show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。