发现大模型中间层隐藏状态能线性解码幻觉信号，定位关键识别层。
Hallucination Is Linearly Decodable from Mid-Layer Hidden States in Quantized LLMs
- 在4比特量化模型中，用线性探测器从中间层提取真相信号。
- 单个中间层探测即达0.904~1.000的AUROC，显著优于采样方法。
- 适用于模型可解释性研究与幻觉检测，尤其适合资源受限场景。
我们研究开源大模型是否在其隐藏状态中编码了可线性分离的真相信号，以及该信号在哪个网络深度最强。在三个7B–8B指令微调模型（Llama-3.1-8B、Mistral-7B、Qwen2.5-7B）上，采用4位NF4量化加载，于四个幻觉基准测试（TruthfulQA、HaluEval-QA、FEVER及可控合成数据集）中提取每层隐藏状态，并比较四种检测方法：线性探针、MLP探针、INSIDE EigenScore、自一致性与注意力熵。在保留分割上，单个中间层的线性探针达到0.904–1.000的AUROC，而基于采样的检测方法未超过0.541 AUROC。真相信号近似线性：MLP探针通常不超出线性探针0.01以上。峰值探测层在自然语言任务中呈现一致区间——对Llama和Mistral为第13–18层（共32层），对Qwen为第19–25层（共28层）。首层注意力熵在知识依赖场景下提供互补信号（在HaluEval-QA上达0.866–0.941 AUROC），且无需额外推理成本。采样方法低表现反映其评估范式与信息访问间的结构性错配，非方法本质缺陷。代码与数据已发布，可在单张8GB GPU上完全复现。
原文摘要 · Abstract (English)
We investigate whether open-source LLMs encode a linearly separable truthfulness signal in their hidden states, and at which network depth this signal is strongest. Across three $7$B--$8$B instruction-tuned models (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B) loaded in $4$-bit NF4 quantization, we extract per-layer hidden states on four hallucination benchmarks (TruthfulQA, HaluEval-QA, FEVER, and a controlled synthetic set) and compare four detection approaches: linear and MLP probes, INSIDE EigenScore, self-consistency, and attention entropy. A linear probe on a single mid-network layer achieves $0.904$--$1.000$ AUROC on held-out splits, while sampling-based detectors do not exceed $0.541$ AUROC under the same protocol. The truthfulness signal is approximately linear: MLP probes rarely surpass linear probes by more than $0.01$ AUROC. Peak probing layers fall in a consistent band across model families on natural-language benchmarks -- blocks~$13$--$18$ of~$32$ for Llama and Mistral, and blocks~$19$--$25$ of~$28$ for Qwen. First-block attention entropy provides a complementary signal in knowledge-grounded settings ($0.866$--$0.941$ AUROC on HaluEval-QA) at no additional inference cost. The low discriminability of sampling methods under this protocol reflects a structural mismatch between paired-label evaluation and the information these methods access, rather than an inherent limitation of those methods. Code and data are released for full reproducibility on a single $8$\,GB GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。