提前识别大模型幻觉并干预，提升回答真实性。
FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs
- 基于模型隐藏状态构建检测器，预测幻觉发生
- 干预后输出事实性提升34.4%，检测准确率超70%
- 轻量级设计，适用于多种大模型和问答场景
语言模型会生成幻觉。本文探究能否在幻觉发生前进行检测与缓解。研究发现，语言模型的内部表示蕴含丰富信号，可用于此目的。我们提出FactCheckmate，通过学习一个分类器，基于输入处理过程中的隐藏状态，在解码开始前预测模型是否会产生幻觉。若检测到幻觉，系统将调整模型隐藏状态，使输出更符合事实。该方法揭示了隐藏状态可反映模型内在机制。实验在不同规模和架构的模型（如Llama、Mistral、Qwen、Gemma）上进行，覆盖多个领域问答数据集。结果表明，FactCheckmate实现超过70%的预判检测准确率；经干预后，输出事实性平均提升34.4%，显著优于多数事后修正方法。其检测与缓解模块轻量，推理开销极小。
原文摘要 · Abstract (English)
Language models (LMs) hallucinate. We inquire: Can we detect and mitigate hallucinations before they happen? This work answers this research question in the positive, by showing that the internal representations of LMs provide rich signals that can be used for this purpose. We introduce FactCheckmate, which preemptively detects hallucinations by learning a classifier that predicts whether the LM will hallucinate, based on the model's hidden states produced over the inputs, before decoding begins. If a hallucination is detected, FactCheckmate then intervenes by adjusting the LM's hidden states such that the model will produce more factual outputs. FactCheckmate provides fresh insights that the inner workings of LMs can be revealed by their hidden states. Practically, both its detection and mitigation models are lightweight, adding little inference overhead; FactCheckmate proves a more efficient approach for mitigating hallucinations compared to many post-hoc alternatives. We evaluate FactCheckmate over LMs of different scales and model families (including Llama, Mistral, Qwen and Gemma), across a variety of QA datasets from different domains. Our results demonstrate the effectiveness of FactCheckmate, achieving over 70% preemptive detection accuracy. On average, outputs generated by LMs with intervention are 34.4% more factual compared to those without.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。