构建首个覆盖多类大模型的摘要幻觉评测集，揭示当前检测模型仍有巨大提升空间。
FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
- 基于10个主流大模型生成摘要，筛选出人类专家标注的难检测幻觉样本
- 即使表现最佳的GPT-3.5-Turbo仍存在幻觉，顶尖检测模型准确率仅约50%
- 适合关注大模型可靠性、幻觉检测与评估方法的研究者使用
摘要生成是大语言模型（LLMs）最常见的任务之一，尤其在检索增强生成（RAG）中广泛应用。然而，现有对大模型摘要幻觉的评估及幻觉检测模型的评测，均因所涉模型种类和家族缺乏多样性和时效性而受限。本文提出FaithBench，一个包含10个现代大模型（来自8个不同家族）生成的挑战性幻觉摘要的评测基准，并由人类专家提供真实标注。所谓‘挑战性’指这些摘要使当前主流、最先进的幻觉检测模型（如GPT-4o-as-a-judge）产生分歧。实验结果表明，GPT-4o与GPT-3.5-Turbo产生的幻觉最少；但即便如此，最优秀的幻觉检测模型在FaithBench上的准确率也仅接近50%，说明该领域仍有巨大改进空间。代码仓库：https://github.com/vectara/FaithBench
原文摘要 · Abstract (English)
Summarization is one of the most common tasks performed by large language models (LLMs), especially in applications like Retrieval-Augmented Generation (RAG). However, existing evaluations of hallucinations in LLM-generated summaries, and evaluations of hallucination detection models both suffer from a lack of diversity and recency in the LLM and LLM families considered. This paper introduces FaithBench, a summarization hallucination benchmark comprising challenging hallucinations made by 10 modern LLMs from 8 different families, with ground truth annotations by human experts. ``Challenging'' here means summaries on which popular, state-of-the-art hallucination detection models, including GPT-4o-as-a-judge, disagreed on. Our results show GPT-4o and GPT-3.5-Turbo produce the least hallucinations. However, even the best hallucination detection models have near 50\% accuracies on FaithBench, indicating lots of room for future improvement. The repo is https://github.com/vectara/FaithBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。