整合29种不确定性量化方法，帮大模型判断何时该信任输出。
UncertaintyZoo: A Unified Toolkit for Quantifying Predictive Uncertainty in Deep Learning Systems
- 统一接口集成29种不确定性量化方法，覆盖5类主流技术。
- 在CodeBERT和ChatGLM3上验证，能有效揭示代码漏洞检测的预测不确定性。
- 适合做安全敏感应用的开发者或研究模型可信度的学者使用。
大型语言模型（LLMs）在问答、自动驾驶、自动软件开发等领域日益广泛应用。尽管取得进展，但作为数据驱动系统，LLMs 经常产生错误预测，在安全关键场景下可能导致严重损失。为衡量模型输出置信度，已有多种不确定性量化（UQ）方法被提出。然而，现有工具匮乏，难以集成这些方法，制约了实际应用与后续研究。为此，本文提出 UncertaintyZoo，一个统一工具包，整合了29种不确定性量化方法，涵盖五大类别，并提供标准化接口。利用 UncertaintyZoo，我们在 CodeBERT 与 ChatGLM3 模型上的代码漏洞检测任务中评估了现有 UQ 方法的有效性。结果表明，UncertaintyZoo 能有效揭示预测中的不确定性。项目演示视频及代码已开源：https://github.com/Paddingbuta/UncertaintyZoo。
原文摘要 · Abstract (English)
Large language models(LLMs) are increasingly expanding their real-world applications across domains, e.g., question answering, autonomous driving, and automatic software development. Despite this achievement, LLMs, as data-driven systems, often make incorrect predictions, which can lead to potential losses in safety-critical scenarios. To address this issue and measure the confidence of model outputs, multiple uncertainty quantification(UQ) criteria have been proposed. However, even though important, there are limited tools to integrate these methods, hindering the practical usage of UQ methods and future research in this domain. To bridge this gap, in this paper, we introduce UncertaintyZoo, a unified toolkit that integrates 29 uncertainty quantification methods, covering five major categories under a standardized interface. Using UncertaintyZoo, we evaluate the usefulness of existing uncertainty quantification methods under the code vulnerability detection task on CodeBERT and ChatGLM3 models. The results demonstrate that UncertaintyZoo effectively reveals prediction uncertainty. The tool with a demonstration video is available on the project site https://github.com/Paddingbuta/UncertaintyZoo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。