构建细粒度对话事实验证基准，提升幻觉检测精度。
FineDialFact: A benchmark for Fine-grained Dialogue Fact Verification
- 将对话回复拆解为原子事实进行逐项验证
- 在开放域数据集上最佳F1仅0.74，任务仍具挑战
- 适合研究幻觉检测与推理模型的学者使用
大语言模型常产生幻觉——即事实性错误或虚构信息，对对话系统等自然语言处理应用构成重大挑战。当前方法多关注生成回复的整体事实一致性，但回复常混合正确、错误与不可验证的事实，单一标签过于粗略。本文提出细粒度对话事实验证基准 FineDialFact，针对从对话回复中提取的原子事实进行验证。基于公开对话数据集构建数据集，并用多种基线方法评估。实验表明，引入思维链（Chain-of-Thought）推理的方法可提升性能，但在 HybriDialogue 开放域数据集上最高 F1 仅为 0.74，说明该任务仍具挑战性。代码与数据已开源于 https://github.com/XiangyanChen/FineDialFact。
原文摘要 · Abstract (English)
Large language models are known to produce hallucinations - factually incorrect or fabricated information - which poses significant challenges for many natural language processing applications, such as dialogue systems. As a result, detecting hallucinations has become a critical area of research. Current approaches to hallucination detection in dialogue systems primarily focus on verifying the factual consistency of generated responses. However, these responses often contain a mix of accurate, inaccurate or non-verifiable facts, making the use of a single factual label overly simplistic and coarse-grained. In this paper, we introduce a benchmark, FineDialFact, for fine-grained dialogue fact verification, which involves verifying atomic facts extracted from dialogue responses. To support this, we construct a dataset based on publicly available dialogue datasets and evaluate it using various baseline methods. Experimental results demonstrate that methods incorporating Chain-of-Thought reasoning can enhance performance in dialogue fact verification. Despite this, the best F1-score achieved on the HybriDialogue, an open-domain dialogue dataset, is only 0.74, indicating that the benchmark remains a challenging task for future research. We release our dataset and code at https://github.com/XiangyanChen/FineDialFact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。