通过分析注意力汇聚点，实现更精准的幻觉检测。
Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
- 利用生成过程中注意力过度集中于某些词的现象作为检测信号。
- 在多个大模型和数据集上达到当前最优的幻觉检测效果。
- 适合关注模型可信性与输出可靠性的研究者使用。
大型语言模型常产生流畅且自信但事实错误的幻觉输出。尽管近期方法尝试从注意力图中提取特征进行幻觉检测，但其背后的机制仍不清晰。本文提出SinkProbe，基于观察：幻觉与注意力‘汇聚点’（attention sinks）密切相关——这些词在生成过程中积累了过量注意力，表明计算从依赖输入的分布式注意力转向依赖先验的压缩式计算。虽然仅通过注意力图计算汇聚分数，但发现分类器主要依赖值向量范数较大的汇聚点。此外，我们证明先前方法在数学上隐含依赖注意力汇聚点。本研究揭示了理论基础并实现了跨主流数据集和大模型的领先检测性能。
原文摘要 · Abstract (English)
Large language models frequently exhibit hallucinations: fluent and confident outputs that are factually incorrect or unsupported by the input context. While recent hallucination detection methods have explored various features derived from attention maps, the underlying mechanisms they exploit remain poorly understood. In this work, we propose SinkProbe, a hallucination detection method grounded in the observation that hallucinations are deeply entangled with attention sinks - tokens that accumulate disproportionate attention mass during generation - indicating a transition from distributed, input-grounded attention to compressed, prior-dominated computation. Importantly, although sink scores are computed solely from attention maps, we find that the classifier preferentially relies on sinks whose associated value vectors have large norms. Moreover, we show that previous methods implicitly depend on attention sinks by establishing their mathematical relationship to sink scores. Our findings yield a novel hallucination detection method grounded in theory that produces state-of-the-art results across popular datasets and LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。