CoT推理虽能减少幻觉,却会隐藏检测信号,让识别更难。
Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation
- 通过实验发现CoT改变模型内部状态和概率分布
- CoT降低幻觉频率但使检测准确率下降15%以上
- 适合研究幻觉检测与推理机制的平衡者阅读
大型语言模型(LLMs)常产生事实错误或语义无关内容,即幻觉。链式思维(CoT) prompting 通过引导分步推理可缓解幻觉,但其对幻觉检测的影响尚不明确。本文开展系统性实证评估:先通过小规模实验揭示CoT显著影响LLM内部状态与词元概率分布;随后在指令微调和推理导向两类主流LLM上,测试多种CoT方法对主流幻觉检测方法的影响。重点考察三个维度:幻觉评分分布变化、检测准确率波动、检测置信度偏移。结果表明,尽管CoT有效降低幻觉发生频率,却同时掩盖了关键检测信号,导致各类检测方法效能下降。本研究揭示了推理应用中被忽视的权衡代价。代码已开源:https://github.com/ECNU-Text-Computing/cot-hallu-detect。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often exhibit \textit{hallucinations}, generating factually incorrect or semantically irrelevant content in response to prompts. Chain-of-Thought (CoT) prompting can mitigate hallucinations by encouraging step-by-step reasoning, but its impact on hallucination detection remains underexplored. To bridge this gap, we conduct a systematic empirical evaluation. We begin with a pilot experiment, revealing that CoT reasoning significantly affects the LLM's internal states and token probability distributions. Building on this, we evaluate the impact of various CoT prompting methods on mainstream hallucination detection methods across both instruction-tuned and reasoning-oriented LLMs. Specifically, we examine three key dimensions: changes in hallucination score distributions, variations in detection accuracy, and shifts in detection confidence. Our findings show that while CoT prompting helps reduce hallucination frequency, it also tends to obscure critical signals used for detection, impairing the effectiveness of various detection methods. Our study highlights an overlooked trade-off in the use of reasoning. Code is publicly available at: https://github.com/ECNU-Text-Computing/cot-hallu-detect .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。