探究大模型不确定性估计的可靠方法，发现压缩特征更抗分布偏移。
From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models
- 分离分析信号设计、训练数据与评估设置对不确定性估计的影响。
- 压缩特征在分布外场景下表现更优，证明仅看域内性能不够。
- 构建可直接使用的基准探针，适合开放生成任务部署。
基于探针的不确定性估计（UE）通过学习模型内部信号来检测大语言模型的幻觉现象，但现有方法在特征设计、训练数据构造和评估设置上同时变化，难以判断性能提升的真实原因。为此，我们提出在一致条件下对探针式UE进行解耦研究。结果表明，原始隐藏状态和注意力特征在域内表现优异，但在分布外场景下，结构化且压缩的特征更具鲁棒性，说明仅依赖域内性能无法衡量进展。此外，提示设计与标签构造显著影响探针行为。基于这些最佳实践，我们训练了可迁移的基准预训练探针，在开放式事实生成任务中表现稳定，提供了一个可靠的开箱即用基线。本工作倡导更面向实际部署的探针式不确定性评估。代码已开源：https://github.com/ponhvoan/ProbeUE。
原文摘要 · Abstract (English)
Probe-based uncertainty estimation (UE) has emerged as a prominent approach to detect hallucinations in Large Language Models (LLMs) by learning uncertainty from internal model signals. Yet, recent methods vary simultaneously across feature design, training data construction, and evaluation setting, obscuring what actually drives performance. To address this issue, we propose a factorised study of probe-based UE under matched conditions. Our results show that raw hidden states and attention features are difficult to outperform in-domain. However, under distribution shift, structured and compressed features are more robust, suggesting that in-domain performance alone is insufficient to measure progress. Furthermore, prompting and label construction significantly affect probe behaviour. Building on these best-practice findings, we train benchmark-based pretrained probes that transfer reasonably well to open-ended factual generation, providing a stable off-the-shelf baseline. Our work encourages more deployment-oriented evaluation of probe-based uncertainty estimators. The code repository is available at https://github.com/ponhvoan/ProbeUE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。