arXiv:2510.04849cs.CL2025-10EMNLP被引 12

构建多语言细粒度幻觉检测数据集,提升大模型事实准确性评估能力

When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA

  • 用三阶段自动化流程生成跨14语言的细粒度幻觉标注数据
  • 编码器模型在多语言上表现最佳,优于基于置信度和LLM标记的方法
  • 数据集支持跨语言泛化,成本远低于人工标注,适合多语言研究者

幻觉检测仍是保障大语言模型安全可靠部署的关键挑战,尤其在需要事实准确性的场景中。现有基准多为英文、仅支持序列级标注,缺乏细粒度与多语言监督。本文提出PsiloQA,一个大规模多语言数据集,覆盖14种语言,标注了细粒度幻觉片段。该数据集通过三阶段自动化流程构建:利用GPT-4o从维基百科生成问答对,让多种LLM在无上下文环境下生成可能幻觉的回答,并用GPT-4o对比黄金答案与检索上下文自动标注幻觉片段。我们评估了多种幻觉检测方法,包括不确定性量化、基于LLM的标记和微调编码器模型,发现编码器模型在所有语言上表现最优。此外,PsiloQA展现出良好的跨语言泛化能力,并可有效迁移至其他基准,且成本显著低于人工标注。本工作推动了多语言场景下可扩展的细粒度幻觉检测发展。

原文摘要 · Abstract (English)

Hallucination detection remains a fundamental challenge for the safe and reliable deployment of large language models (LLMs), especially in applications requiring factual accuracy. Existing hallucination benchmarks often operate at the sequence level and are limited to English, lacking the fine-grained, multilingual supervision needed for a comprehensive evaluation. In this work, we introduce PsiloQA, a large-scale, multilingual dataset annotated with span-level hallucinations across 14 languages. PsiloQA is constructed through an automated three-stage pipeline: generating question-answer pairs from Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse LLMs in a no-context setting, and automatically annotating hallucinated spans using GPT-4o by comparing against golden answers and retrieved context. We evaluate a wide range of hallucination detection methods -- including uncertainty quantification, LLM-based tagging, and fine-tuned encoder models -- and show that encoder-based models achieve the strongest performance across languages. Furthermore, PsiloQA demonstrates effective cross-lingual generalization and supports robust knowledge transfer to other benchmarks, all while being significantly more cost-efficient than human-annotated datasets. Our dataset and results advance the development of scalable, fine-grained hallucination detection in multilingual settings.

幻觉检测多语言细粒度数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。