模型明明知道正确答案，却因微小扰动产生高自信错误回答。
Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer
- 发现一种新幻觉：模型在确定正确答案时仍会因微小变化出错
- 该现象在不同模型和数据集上稳定存在，现有方法反而更差
- 提出基于探测的新方法，在高可信幻觉场景表现更优
以往研究将大语言模型（LLM）幻觉归因于不确定性或知识错误。本文定义并研究一种新型幻觉——模型能持续正确回答问题，但现实中常见的微小扰动会导致其以高置信度生成幻觉响应。这种现象被称为CHOKE（高可信幻觉覆盖已知证据）。该现象在医疗、法律等高风险领域尤为危险，因模型的高置信度常被误用为可靠性指标。我们发现，CHOKE 示例在不同提示下具有一致性，存在于多种模型与数据集，且与传统幻觉本质不同，导致现有缓解方法在处理此类幻觉时表现更差。最后，我们提出一种基于探测的缓解策略，在应对 CHOKE 幻觉上优于现有方法。研究揭示了幻觉的一个被忽视维度，强调需深入理解其成因并改进缓解策略以提升 LLM 安全性。代码已开源。
原文摘要 · Abstract (English)
Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge. In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty. This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability. We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations. This difference leads existing mitigation methods to perform worse on CHOKE examples than on general hallucinations. Finally, we introduce a probing-based mitigation that outperforms existing methods on CHOKE hallucinations. These findings reveal an overlooked aspect of hallucinations, emphasizing the need to understand their origins and improve mitigation strategies to enhance LLM safety. The code is available at https://github.com/technion-cs-nlp/Trust_me_Im_wrong .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。