探究推理模型是否更易幻觉,发现训练方式决定事实性
Are Reasoning Models More Prone to Hallucination?
- 通过全周期评估发现,冷启动微调+可验证奖励强化能有效降低幻觉
- 两种关键认知缺陷:错误重复与思维-答案不一致会加剧幻觉
- 模型不确定性和事实准确性错位是幻觉根源,适合对齐研究者参考
近期发展的大模型推理能力(LRMs)在复杂任务中表现优异,具备长链式思维(CoT)能力。由于这些模型主要通过后训练在形式化推理任务上优化,其推理能力能否泛化以减少事实类任务中的幻觉仍不明确且存在争议。例如,DeepSeek-R1在SimpleQA基准上性能提升,而OpenAI-o3却出现更严重的幻觉。本文从三个角度展开研究:(1)对LRMs的幻觉进行整体评估,发现经历完整后训练流程(冷启动监督微调SFT + 可验证奖励强化学习)的模型幻觉显著降低;而仅靠蒸馏或无冷启动微调的强化学习则引入更细微的幻觉。(2)行为分析揭示两种直接影响事实性的认知行为:错误重复(表面推理反复沿用同一错误逻辑)和思维-答案不一致(最终答案未忠实匹配前序推理过程)。(3)从模型不确定性视角分析,发现幻觉增加通常与模型不确定性与事实准确性的错位相关。本工作为理解LRMs中的幻觉提供了初步框架。
原文摘要 · Abstract (English)
Recently evolved large reasoning models (LRMs) show powerful performance in solving complex tasks with long chain-of-thought (CoT) reasoning capability. As these LRMs are mostly developed by post-training on formal reasoning tasks, whether they generalize the reasoning capability to help reduce hallucination in fact-seeking tasks remains unclear and debated. For instance, DeepSeek-R1 reports increased performance on SimpleQA, a fact-seeking benchmark, while OpenAI-o3 observes even severer hallucination. This discrepancy naturally raises the following research question: Are reasoning models more prone to hallucination? This paper addresses the question from three perspectives. (1) We first conduct a holistic evaluation for the hallucination in LRMs. Our analysis reveals that LRMs undergo a full post-training pipeline with cold start supervised fine-tuning (SFT) and verifiable reward RL generally alleviate their hallucination. In contrast, both distillation alone and RL training without cold start fine-tuning introduce more nuanced hallucinations. (2) To explore why different post-training pipelines alters the impact on hallucination in LRMs, we conduct behavior analysis. We characterize two critical cognitive behaviors that directly affect the factuality of a LRM: Flaw Repetition, where the surface-level reasoning attempts repeatedly follow the same underlying flawed logic, and Think-Answer Mismatch, where the final answer fails to faithfully match the previous CoT process. (3) Further, we investigate the mechanism behind the hallucination of LRMs from the perspective of model uncertainty. We find that increased hallucination of LRMs is usually associated with the misalignment between model uncertainty and factual accuracy. Our work provides an initial understanding of the hallucination in LRMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。