为大模型幻觉检测提供可定制的置信度评分工具包
Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers
- 将多种不确定性量化方法转为0-1的置信度评分
- 可调参集成多个评分，显著提升幻觉检测效果
- 适合医疗、金融等高风险场景的可靠性增强
大语言模型的幻觉问题在医疗、金融等高风险领域尤为突出。本文提出一个通用的闭卷幻觉检测框架，整合黑盒、白盒及大模型作为裁判（LLM-as-a-Judge）等多种不确定性量化技术，将其统一转换为0到1之间的响应级置信度分数。通过可调参的集成策略，支持任意组合不同分数来源，以适配特定应用场景。实验基于多个问答基准测试，结果表明该集成方法普遍优于各单项组件及现有方法。配套开源工具UQLM提供了完整实现，便于实际部署。研究验证了定制化幻觉检测策略对提升大模型准确性与可靠性的重要价值。
原文摘要 · Abstract (English)
Hallucinations are a persistent problem with Large Language Models (LLMs). As these models become increasingly used in high-stakes domains, such as healthcare and finance, the need for effective hallucination detection is crucial. To this end, we outline a versatile framework for closed-book hallucination detection that practitioners can apply to real-world use cases. To achieve this, we adapt a variety of existing uncertainty quantification (UQ) techniques, including black-box UQ, white-box UQ, and LLM-as-a-Judge, transforming them as necessary into standardized response-level confidence scores ranging from 0 to 1. To enhance flexibility, we propose a tunable ensemble approach that incorporates any combination of the individual confidence scores. This approach enables practitioners to optimize the ensemble for a specific use case for improved performance. To streamline implementation, the full suite of scorers is offered in this paper's companion Python toolkit, UQLM. To evaluate the performance of the various scorers, we conduct an extensive set of experiments using several LLM question-answering benchmarks. We find that our tunable ensemble typically surpasses its individual components and outperforms existing hallucination detection methods. Our results demonstrate the benefits of customized hallucination detection strategies for improving the accuracy and reliability of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。