用单条生成结果高效估算大模型输出的不确定性。
Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure
- 基于最优序列的负对数似然，构建理论严谨的不确定性度量。
- 仅需贪婪解码一次即可计算，性能媲美多序列方法。
- 适合需要快速评估生成质量的部署场景。
大型语言模型在现实应用中日益普及，对其生成文本可信度的评估变得至关重要。可靠的不确定性估计是关键。现有主流方法需生成并分析多个输出序列，计算成本高且难以规模化。本文从适当评分规则框架出发，发现最可能输出序列的负对数似然是一个理论上合理的不确定性度量。为此提出G-NLL方法，仅需贪婪解码得到的一条序列即可近似该度量，显著提升效率并保持理论一致性。实验表明，G-NLL在多种场景下达到当前最优性能。本工作为自然语言生成中的高效可靠不确定性估计奠定了理论基础，挑战了复杂且资源密集的主流方法的必要性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. To this end, reliable uncertainty estimation is essential. Leading uncertainty estimation methods generate and analyze multiple output sequences, which is computationally expensive and impractical at scale. In this work, we inspect the theoretical foundations of these methods and explore new directions to enhance computational efficiency. Building on the framework of proper scoring rules, we find that the negative log-likelihood of the most likely output sequence constitutes a theoretically principled uncertainty measure. To approximate this alternative measure, we propose G-NLL, obtained using a single output sequence from greedy decoding. This approach streamlines uncertainty estimation while preserving theoretical rigor. Empirical results demonstrate that G-NLL achieves state-of-the-art performance across various scenarios. Our work lays the theoretical foundation for efficient and reliable uncertainty estimation in natural language generation, challenging the necessity of the prevalent methods that are more complex and resource-intensive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。