让大模型推理更可信，通过逐步奖励减少幻觉。
FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximization
- 设计逐步奖励机制，惩罚无依据推理步骤。
- 在多个模型和任务上降低幻觉率，正确率不降反升。
- 适合需要可靠逻辑推理的场景，如数学、法律分析。
基于可验证奖励的强化学习（RLVR）显著提升了大语言模型在多步推理任务中的表现。然而，多数RLVR流程依赖稀疏的结果奖励，对中间步骤监督不足，导致模型过度自信和虚假推理，从而增加幻觉。为此，我们提出FaithRL，一种通用强化学习框架，直接优化推理可信度。我们形式化了可信度最大化目标，并理论上证明优化该目标可缓解过度自信。为实现此目标，引入几何奖励设计和可信度感知的优势调节机制，通过惩罚无支持步骤而保留有效部分推导，实现逐步信用分配。在多种模型和基准测试中，FaithRL持续降低幻觉率，同时保持（或提升）答案正确性。进一步分析证实，FaithRL增强了逐步推理的可信度并具备良好泛化能力。代码已开源：https://github.com/aintdoin/FaithRL。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has markedly improved the performance of Large Language Models (LLMs) on tasks requiring multi-step reasoning. However, most RLVR pipelines rely on sparse outcome-based rewards, providing little supervision over intermediate steps and thus encouraging over-confidence and spurious reasoning, which in turn increases hallucinations. To address this, we propose FaithRL, a general reinforcement learning framework that directly optimizes reasoning faithfulness. We formalize a faithfulness-maximization objective and theoretically show that optimizing it mitigates over-confidence. To instantiate this objective, we introduce a geometric reward design and a faithfulness-aware advantage modulation mechanism that assigns step-level credit by penalizing unsupported steps while preserving valid partial derivations. Across diverse backbones and benchmarks, FaithRL consistently reduces hallucination rates while maintaining (and often improving) answer correctness. Further analysis confirms that FaithRL increases step-wise reasoning faithfulness and generalizes robustly. Our code is available at https://github.com/aintdoin/FaithRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。