提升大模型生成结果的可信度，通过校准不确定性得分。
Towards Harmonized Uncertainty Estimation for Large Language Models
- 用轻量级模型校准大模型的不确定性得分
- 在多个任务上最高提升60%的估计准确性
- 适合需要可靠输出的部署场景
为实现大语言模型（LLMs）的稳健与可信部署，量化其生成结果的可靠性至关重要。尽管近期研究利用大模型内部逻辑和语言特征估算不确定性，但我们的实证分析揭示了这些方法在指示性、平衡性和校准性之间难以协调的问题，限制了其准确估算能力。为此，我们提出CUE（Uncertainty Estimation Corrector）：一种基于目标大模型性能对齐数据训练的轻量级模型，用于修正不确定性得分。在多种模型和任务上的全面实验表明，该方法相较于现有方法可实现最高达60%的一致性提升。
原文摘要 · Abstract (English)
To facilitate robust and trustworthy deployment of large language models (LLMs), it is essential to quantify the reliability of their generations through uncertainty estimation. While recent efforts have made significant advancements by leveraging the internal logic and linguistic features of LLMs to estimate uncertainty scores, our empirical analysis highlights the pitfalls of these methods to strike a harmonized estimation between indication, balance, and calibration, which hinders their broader capability for accurate uncertainty estimation. To address this challenge, we propose CUE (Corrector for Uncertainty Estimation): A straightforward yet effective method that employs a lightweight model trained on data aligned with the target LLM's performance to adjust uncertainty scores. Comprehensive experiments across diverse models and tasks demonstrate its effectiveness, which achieves consistent improvements of up to 60% over existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。