用简单线性模型提升大模型不确定性估计的准确性
Can Linear Probes Measure LLM Uncertainty?
- 基于贝叶斯线性回归建模层间特征关系
- 在多个大模型上显著优于现有最优基线
- 方法轻量高效，适合部署于实际系统
有效的不确定性量化（UQ）是大型语言模型在自动决策等场景中可靠应用的关键。然而，在多选结构生成任务中，当前最优的UQ方法仍停留在仅使用最大Softmax得分的朴素基线。为解决这一问题，我们证明：即使采用最简单的线性回归模型，通过贝叶斯统计的严谨方法也能实现性能提升。具体地，我们训练多个贝叶斯线性模型，每个模型预测某一层的输出由前一层输出决定。基于得到的层级后验分布，通过识别稀疏的分布特征组合，推断出模型全局不确定性，形成高效的UQ方案。在多个LLM上的数值实验表明，该方法持续优于现有最优基线。
原文摘要 · Abstract (English)
Effective Uncertainty Quantification (UQ) represents a key aspect for reliable deployment of Large Language Models (LLMs) in automated decision-making and beyond. Yet, for LLM generation with multiple choice structure, the state-of-the-art in UQ is still dominated by the naive baseline given by the maximum softmax score. To address this shortcoming, we demonstrate that taking a principled approach via Bayesian statistics leads to improved performance despite leveraging the simplest possible model, namely linear regression. More precisely, we propose to train multiple Bayesian linear models, each predicting the output of a layer given the output of the previous one. Based on the obtained layer-level posterior distributions, we infer the global uncertainty level of the LLM by identifying a sparse combination of distributional features, leading to an efficient UQ scheme. Numerical experiments on various LLMs show consistent improvement over state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。