让大模型推理更准又少胡说，通过步步验证事实来优化训练。
Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
- 每步推理都自动核对证据，动态调整奖励以强化事实正确性。
- 在数学和幻觉测试中，幻觉率下降37%，准确率提升12%。
- 适合需要高可信度推理的科研、医疗等严谨场景使用。
大型语言模型（LLMs）通过强化学习（RL）优化，在推理任务上取得显著进展，广泛应用于各类挑战性基准测试。然而，我们的实证分析揭示了一个关键缺陷：面向推理的强化学习微调会显著增加幻觉现象。理论分析表明，高方差梯度、熵引起的随机性以及对虚假局部最优解的敏感性是导致幻觉的主要因素。为此，我们提出事实感知分步策略优化（FSPO），一种创新的强化学习微调算法，在每个推理步骤中引入显式事实验证。FSPO利用自动化验证机制对比给定证据，动态调整逐标记的优势值，从而在整个推理过程中激励事实准确性。在Qwen2.5与Llama模型上，基于数学推理与幻觉基准的实验表明，FSPO能有效降低幻觉现象，同时提升推理准确率，显著增强模型的可靠性与性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。