增强大模型推理能力反而加剧工具幻觉,需重新设计训练目标。
The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination
- 通过强化学习逐步提升推理,工具幻觉随性能提高而上升。
- 即使在非工具任务上训练,也会放大后续工具幻觉。
- 现有方法无法兼顾能力与可靠性,存在根本性权衡。
提升大型语言模型(LLMs)的推理能力是构建“先思考后行动”智能体的关键策略。然而,近期观察发现如OpenAI的o3模型,更强推理常伴随更高幻觉率,但尚无研究系统检验推理增强是否直接导致工具幻觉。为此,我们提出SimpleToolHalluBench诊断基准,用于衡量两种失效模式下的工具幻觉:(i) 无可用工具;(ii) 仅有干扰工具可用。通过受控实验,我们得出三个核心发现:第一,证实因果关系——通过强化学习逐步增强推理,工具幻觉与任务性能提升成比例增长;第二,该效应超越过拟合:在非工具任务(如数学)上训练仍会显著加剧后续工具幻觉;第三,该效应具有方法无关性——无论通过监督微调注入推理,还是在推理时通过思维链提示激发推理,均出现同样现象。我们还评估了提示工程和直接偏好优化(DPO)等缓解策略,揭示出可靠性和能力之间存在根本性权衡:降低幻觉必然损害实用性。机制分析表明,推理强化会不成比例地压缩与工具可靠性相关的表征,幻觉表现为晚期残差流中的放大偏差。这些发现表明,当前推理增强方法本身就会放大工具幻觉,亟需新的联合优化能力与可靠性的训练目标。
原文摘要 · Abstract (English)
Enhancing the reasoning capabilities of Large Language Models (LLMs) is a key strategy for building Agents that "think then act." However, recent observations, like OpenAI's o3, suggest a paradox: stronger reasoning often coincides with increased hallucination, yet no prior work has systematically examined whether reasoning enhancement itself causes tool hallucination. To address this gap, we pose the central question: Does strengthening reasoning increase tool hallucination? To answer this, we introduce SimpleToolHalluBench, a diagnostic benchmark measuring tool hallucination in two failure modes: (i) no tool available, and (ii) only distractor tools available. Through controlled experiments, we establish three key findings. First, we demonstrate a causal relationship: progressively enhancing reasoning through RL increases tool hallucination proportionally with task performance gains. Second, this effect transcends overfitting - training on non-tool tasks (e.g., mathematics) still amplifies subsequent tool hallucination. Third, the effect is method-agnostic, appearing when reasoning is instilled via supervised fine-tuning and when it is merely elicited at inference by switching from direct answers to step-by-step thinking. We also evaluate mitigation strategies including Prompt Engineering and Direct Preference Optimization (DPO), revealing a fundamental reliability-capability trade-off: reducing hallucination consistently degrades utility. Mechanistically, Reasoning RL disproportionately collapses tool-reliability-related representations, and hallucinations surface as amplified divergences concentrated in late-layer residual streams. These findings reveal that current reasoning enhancement methods inherently amplify tool hallucination, highlighting the need for new training objectives that jointly optimize for capability and reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。