让小模型推理更真实，避免因答对而奖励错误步骤
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
- 用过程奖励模型给每一步推理打分，显式鼓励真实推理路径
- 在多个小模型和开放问答数据集上，显著减少中间步骤幻觉
- 适合资源受限场景下需要可靠推理的小模型应用
随着大语言模型向小型化、高效化发展，小推理模型（SRMs）在资源受限环境下实现思维链（CoT）推理至关重要。然而，它们容易在中间推理步骤中产生忠实性幻觉。现有基于在线强化学习的缓解方法依赖结果奖励或粗粒度的CoT评估，当最终答案正确时，可能无意中强化了不忠实的推理过程。为此，我们提出忠实感知的逐步强化学习（FaithRL），通过来自过程奖励模型的显式忠实性奖励实现逐步监督，并结合隐式截断重采样策略，从忠实前缀生成对比信号，同时缓解逐步奖励带来的奖励劫持问题。在多个SRM和开放问答基准上的实验表明，FaithRL能持续减少CoT及最终答案中的幻觉，提升推理的忠实性与可靠性。代码已开源。
原文摘要 · Abstract (English)
As large language models become smaller and more efficient, small reasoning models (SRMs) are crucial for enabling chain-of-thought (CoT) reasoning in resource-constrained settings. However, they are prone to faithfulness hallucinations, especially in intermediate reasoning steps. Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained CoT evaluation, which can inadvertently reinforce unfaithful reasoning when the final answer is correct. To address these limitations, we propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model, together with an implicit truncated resampling strategy that generates contrastive signals from faithful prefixes, while also mitigating reward hacking from step-level rewards. Experiments across multiple SRMs and Open-Book QA benchmarks demonstrate that FaithRL consistently reduces hallucinations in both the CoT and final answers, leading to more faithful and reliable reasoning. Code is available at https://github.com/Easy195/FaithRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。