用小型开源模型分析大模型输出,高效检测幻觉内容。
Hallucination Detection via Activations of Open-Weight Proxy Analyzers

- 通过小模型读取生成文本并分析其内部激活,间接检测幻觉。
- 在多个数据集上超越现有方法,最大提升10.3个百分点。
- 模型越大越强不成立,小模型表现甚至更优,适合资源有限场景。
我们提出一种代理分析器框架,用于检测大语言模型中的幻觉。系统不依赖生成模型内部结构,而是通过一个本地部署的小型开源模型读取已生成文本,并利用该模型的内部激活捕捉幻觉。此方法对闭源API(如GPT-4)和任意开源模型均有效。我们构建了18个基于Transformer文本处理机制的特征,涵盖残差流范数、各注意力头的文档注意力、熵、MLP激活、逻辑光谱轨迹及三项新的词元级事实锚定统计量。在5个幻觉数据集共72,135样本上训练堆叠集成模型。测试覆盖7种分析器架构(0.5亿至90亿参数):Qwen2.5(0.5B与7B)、Gemma-2(2B与9B)、Pythia(1.4B)、LLaMA-3(3B与8B)。所有模型在RAGTruth任务上均优于ReDeEP的0.73 AUC,提升7.4至10.3个百分点;其中Qwen2.5-7B达到0.717 F1,略超ReDeEP的0.713,而Qwen2.5-0.5B达0.706。最显著发现是七种模型表现高度集中,AUC差异仅2.3个百分点,跨越18倍参数规模。更意外的是,3B版LLaMA在RAGTruth上表现优于8B版,表明同一模型家族中并非越大越好。两个测试数据集均包含多类大模型输出,结果非特定生成器偏差。
原文摘要 · Abstract (English)
We introduce a proxy-analyzer framework for detecting hallucinations in large language models. Instead of looking inside the generating model, our system reads already-generated text through a small locally hosted open-weight model and spots hallucinations using the reader's own internal activations. This works just as well when the generator is a closed API like GPT-4 as when it is any open-weight model. We built eighteen features grounded in how transformers process text, covering residual stream norms, per-head source-document attention, entropy, MLP activations, logit-lens trajectories, and three new token-level grounding statistics. We trained a stacking ensemble on 72,135 samples from five hallucination datasets. We tested across seven analyzer architectures from 0.5 billion to 9 billion parameters: Qwen2.5 at 0.5B and 7B, Gemma-2 at 2B and 9B, Pythia at 1.4B, and LLaMA-3 at both 3B and 8B. Across all seven, we consistently beat ReDeEP's token-level AUC of 0.73 on RAGTruth by 7.4 to 10.3 percentage points. Qwen2.5-7B reached an F1 of 0.717, just above ReDeEP's 0.713, while Qwen2.5-0.5B hit 0.706. The most striking finding is how tightly all seven models cluster: AUC spans only 2.3 percentage points across an eighteen-fold difference in model size. Even more surprising, our 3B LLaMA outperforms our 8B LLaMA on RAGTruth, showing that bigger is not always better even within the same model family. Both RAGTruth and LLM-AggreFact include outputs from multiple LLM families, so our results are not skewed toward any particular generator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。