用少量标注数据提升大模型置信度估计的准确性。
Improved Confidence Estimates for Black-Box Large Language Models
- 基于相似查询正确性构建简单分类器优化置信度
- 在真实数据集上显著优于现有零样本方法
- 计算开销极小，适合实际部署场景
不确定性量化（UQ）对大语言模型（LLM）的安全部署至关重要。现有方法（从口语化置信度到需多次生成的方法）通常为零样本，无需标签数据即可生成不确定性评分。然而，在实际应用中，必须在目标数据集上评估其性能后才能部署。本文表明，通过利用该数据集，我们能持续超越现有评分方法。具体而言，我们构建了简单的分类器，以这些评分和相似查询的正确性作为特征，预测LLM输出的正确性。该方法计算开销极小，可作为大模型不确定性量化在现实应用中的低成本、易实现增强方案。
原文摘要 · Abstract (English)
Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。