构建实体级幻觉检测数据集，揭示大模型幻觉定位难题
HalluEntity: Benchmarking and Understanding Entity-Level Hallucination Detection
- 提出实体级幻觉标注数据集HalluEntity，精准定位幻觉源头
- 17个主流大模型测试显示，基于概率的检测方法误报率高
- 发现语言特性与幻觉倾向相关，为未来研究指明方向
为缓解大模型幻觉问题，现有研究多通过不确定性估计检测幻觉，但主要集中在句子或段落级别，难以精确定位具体实体。针对长文本中真假信息混杂的问题，本文提出实体级幻觉检测新范式。构建了首个实体级幻觉标注数据集HalluEntity，涵盖多种类型实体。基于该数据集，系统评估了17个现代大模型在不确定性估计方法上的表现。实验结果表明，仅依赖单个词元概率的方法易过度预测幻觉，而上下文感知方法虽有改进但仍不理想。进一步定性分析揭示幻觉倾向与语言特征存在关联，指出未来研究关键方向。
原文摘要 · Abstract (English)
To mitigate the impact of hallucination nature of LLMs, many studies propose detecting hallucinated generation through uncertainty estimation. However, these approaches predominantly operate at the sentence or paragraph level, failing to pinpoint specific spans or entities responsible for hallucinated content. This lack of granularity is especially problematic for long-form outputs that mix accurate and fabricated information. To address this limitation, we explore entity-level hallucination detection. We propose a new data set, HalluEntity, which annotates hallucination at the entity level. Based on the dataset, we comprehensively evaluate uncertainty-based hallucination detection approaches across 17 modern LLMs. Our experimental results show that uncertainty estimation approaches focusing on individual token probabilities tend to over-predict hallucinations, while context-aware methods show better but still suboptimal performance. Through an in-depth qualitative study, we identify relationships between hallucination tendencies and linguistic properties and highlight important directions for future research. HalluEntity: https://huggingface.co/datasets/samuelyeh/HalluEntity
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。