通过语义传播与生成中的不确定性重校困惑度，实现可解释的幻觉检测。
RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detection
- 基于语义传播和语言生成的双重不确定性重构困惑度
- 在多个QA数据集上平均AUC达0.833，性能最优
- 输出每个词的不确定得分，可定位幻觉触发源
大型语言模型虽强大，但幻觉仍是可信应用的主要障碍。现有方法通过测量不确定性提升幻觉检测能力，却无法解释幻觉发生的根源，尤其难以识别哪些输入部分易引发幻觉。近期研究指出，注意力机制在层间逐步融合局部标记信息形成高层语义时存在不确定性（语义传播），同时语言生成因基于概率选择高层语义而产生不确定性。为此，本文提出RePPL，从这两个方面重新校准不确定性度量，将不确定性分数分配给每个词，并以类似困惑度的对数平均形式聚合为总分。实验表明，该方法在多个QA数据集上对先进模型的综合检测性能最佳（平均AUC 0.833），且能生成词级不确定性分数，解释幻觉成因。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become powerful, but hallucinations remain a vital obstacle to their trustworthy use. Previous works improved the capability of hallucination detection by measuring uncertainty. But they can not explain the provenance behind why hallucinations occur, particularly in identifying which part of the inputs tends to trigger hallucinations. Recent works on the prompt attack indicate that uncertainty exists in semantic propagation, where attention mechanisms gradually fuse local token information into high-level semantics across layers. Meanwhile, uncertainty also emerges in language generation, due to its probability-based selection of high-level semantics for sampled generations. Based on that, we propose RePPL to recalibrate uncertainty measurement by these two aspects, which dispatches explainable uncertainty scores to each token and aggregates in Perplexity-style Log-Average form as a total score. Experiments show that it achieves the best comprehensive detection performance across various QA datasets on advanced models (average AUC of 0.833), and it is capable of producing token-level uncertainty scores as explanations of hallucination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。