提出长度无关的不确定性评估方法，提升大模型输出可靠性判断。
UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Models
- 通过残差回归消除输出长度对不确定性的干扰
- 在多个任务中显著优于现有长度归一化方法
- 无需修改模型，可通用适配各类不确定性评估指标
大型语言模型已成为众多应用的核心工具，确保其输出质量和可信度至关重要。这推动了对输出可靠性进行不确定性量化（UQ）方法的研究。现有许多UQ技术依赖于词元概率，但会引入输出长度相关的偏差。尽管部分方法尝试校正此问题，我们仍发现即使在长度归一化方法中，偏差依然存在。为此，本文提出UNCERTAINTY-LINE：一种简单的去偏方法，通过将不确定性分数对输出长度进行回归，并使用残差作为修正后的、长度无关的估计值。该方法为后处理、模型无关，适用于多种UQ度量。在机器翻译、摘要生成和问答任务上的大量实验表明，UNCERTAINTY-LINE在多个指标和模型上均持续优于现有的名义长度归一化方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become indispensable tools across various applications, making it more important than ever to ensure the quality and the trustworthiness of their outputs. This has led to growing interest in uncertainty quantification (UQ) methods for assessing the reliability of LLM outputs. Many existing UQ techniques rely on token probabilities, which inadvertently introduces a bias with respect to the length of the output. While some methods attempt to account for this, we demonstrate that such biases persist even in length-normalized approaches. To address the problem, here we propose UNCERTAINTY-LINE: (Length-INvariant Estimation), a simple debiasing procedure that regresses uncertainty scores on output length and uses the residuals as corrected, length-invariant estimates. Our method is post-hoc, model-agnostic, and applicable to a range of UQ measures. Through extensive evaluation on machine translation, summarization, and question-answering tasks, we demonstrate that UNCERTAINTY-LINE: consistently improves over even nominally length-normalized UQ methods uncertainty estimates across multiple metrics and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。