通过分析模型内部状态，揭示幻觉生成原因并实现无需外部数据的检测。
What are Models Thinking about? Understanding Large Language Model Hallucinations "Psychology" through Model Inner State Analysis
- 分推理三阶段提取内部状态，追踪幻觉生成过程。
- 不依赖外部知识源，在多个数据集上实现高精度幻觉检测。
- 为理解大模型幻觉机制提供可解释性工具，适合模型安全研究者。
大型语言模型（LLM）在生成内容时存在事实性不一致问题，导致幻觉生成。现有幻觉检测方法多依赖外部信息源（如RAG），引入额外延迟。近年来，模型推理过程中的内部状态被广泛用于各类研究，如提示注入检测等。鉴于其可解释性且无需外部信息，本文将内部状态引入幻觉检测任务。系统分析了不同内部状态在推理前向过程中的表现特征，并在理解、查询、生成三个阶段分别提取状态信息。通过分析这些状态，深入揭示了幻觉内容生成的内在机理。进一步将内部状态用于幻觉检测，开展全面实验，验证其优势与局限性。
原文摘要 · Abstract (English)
Large language model (LLM) systems suffer from the models' unstable ability to generate valid and factual content, resulting in hallucination generation. Current hallucination detection methods heavily rely on out-of-model information sources, such as RAG to assist the detection, thus bringing heavy additional latency. Recently, internal states of LLMs' inference have been widely used in numerous research works, such as prompt injection detection, etc. Considering the interpretability of LLM internal states and the fact that they do not require external information sources, we introduce such states into LLM hallucination detection. In this paper, we systematically analyze different internal states' revealing features during inference forward and comprehensively evaluate their ability in hallucination detection. Specifically, we cut the forward process of a large language model into three stages: understanding, query, generation, and extracting the internal state from these stages. By analyzing these states, we provide a deep understanding of why the hallucinated content is generated and what happened in the internal state of the models. Then, we introduce these internal states into hallucination detection and conduct comprehensive experiments to discuss the advantages and limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。