构建统一基准,系统评估大模型幻觉检测在多场景下的表现
OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

- 建立标准化评估流程,覆盖提示构造到真伪标注全流程
- 支持黑箱、灰箱、白箱三类检测方法,实现跨范式对比
- 开源代码与数据集,助力可复现的幻觉检测研究
幻觉检测对大语言模型的可靠部署至关重要。现有评估面临两个核心挑战:推理配置与评价标准不一致,下游任务与领域覆盖有限。导致报告的检测性能难以比较、复现和泛化。我们提出 OpenHalDet,一个面向多样化生成场景的统一幻觉检测基准。该基准标准化了从提示构建、响应生成、真实性标注、检测器评分到指标计算的完整评估流程。支持不同访问方式下的异构检测器家族:仅使用生成输出的黑箱方法、依赖概率信号的灰箱方法,以及利用内部模型信号的白箱方法。通过将多样任务、模型与检测器纳入统一框架,OpenHalDet 实现了受控对比,并系统揭示不同检测范式在大模型应用中的行为特征。我们开放发布 OpenHalDet 的代码库,支持可复现评估与未来幻觉检测方法的持续发展。代码与数据集详见 https://github.com/Nellie179/Hallucination-Detection。
原文摘要 · Abstract (English)
Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challenges: inconsistent inference configuration and evaluation, and limited coverage of downstream domains and tasks. Consequently, reported detector performance is often difficult to compare, reproduce, and generalize beyond specific experimental settings. We introduce OpenHalDet, a unified benchmark for hallucination detection across diverse generation scenarios. OpenHalDet standardizes the evaluation pipeline, from prompt construction and response generation to truthfulness annotation, detector scoring, and metric computation. It supports heterogeneous detector families under different access settings, including black-box methods that use only generated outputs, gray-box methods that rely on probability-based signals, and white-box methods that exploit internal model signals. By bringing diverse tasks, models, and detectors into a shared framework, OpenHalDet enables controlled comparison and provides a systematic view of how different detection paradigms behave in LLM applications. We release OpenHalDet as an open and extensible codebase to facilitate reproducible evaluation and future development of hallucination detection methods. The code and datasets are available at https://github.com/Nellie179/Hallucination-Detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。