用词频覆盖度检测大模型幻觉,发现能辅助提升判断准确率。
Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models
- 构建红辣椒数据集的后缀数组,统计问题与生成文本的词组出现频率。
- 单独使用词频特征效果弱,但与概率值结合后在高不确定性数据集上提升检测效果。
- 适合关注大模型可靠性、幻觉检测与训练数据影响的研究者。
大语言模型在开放域问答中的幻觉问题是一个根本性挑战。以往研究依赖模型内部信号(如标记级熵或生成一致性)检测幻觉,而预训练数据暴露与幻觉之间的关系尚未深入探索。已有研究表明,模型在长尾知识上表现较差——对于预训练中罕见的实体,生成答案的准确率下降。然而,数据覆盖率本身能否作为检测信号仍被忽视。本文提出一个补充性问题:问题和/或生成答案的词汇训练数据覆盖率是否可提供额外的幻觉检测信号?为此,我们在1.3万亿标记的RedPajama预训练语料库上构建可扩展的后缀数组,以获取提示词与模型生成内容的n-gram统计信息,并在三个QA基准上评估其检测有效性。结果表明,虽然基于出现次数的特征单独使用时预测能力较弱,但在与对数概率结合后,尤其在内在模型不确定性较高的数据集上,能带来适度提升。这说明词汇覆盖率可为幻觉检测提供互补信号。所有代码与后缀数组基础设施已开源于https://github.com/WWWonderer/ostd。
原文摘要 · Abstract (English)
Hallucination in large language models (LLMs) is a fundamental challenge, particularly in open-domain question answering. Prior work attempts to detect hallucination with model-internal signals such as token-level entropy or generation consistency, while the connection between pretraining data exposure and hallucination is underexplored. Existing studies show that LLMs underperform on long-tail knowledge, i.e., the accuracy of the generated answer drops for the ground-truth entities that are rare in pretraining. However, examining whether data coverage itself can serve as a detection signal is overlooked. We propose a complementary question: Does lexical training-data coverage of the question and/or generated answer provide additional signal for hallucination detection? To investigate this, we construct scalable suffix arrays over RedPajama's 1.3-trillion-token pretraining corpus to retrieve $n$-gram statistics for both prompts and model generations. We evaluate their effectiveness for hallucination detection across three QA benchmarks. Our observations show that while occurrence-based features are weak predictors when used alone, they yield modest gains when combined with log-probabilities, particularly on datasets with higher intrinsic model uncertainty. These findings suggest that lexical coverage features provide a complementary signal for hallucination detection. All code and suffix-array infrastructure are provided at https://github.com/WWWonderer/ostd.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。