找出了与人类不确定性更匹配的模型不确定度衡量方法。
Investigating Human-Aligned Large Language Model Uncertainty
- 对比多种不确定度指标，发现贝叶斯和top-k熵更贴合人类判断。
- 大模型下部分指标反而偏离人类认知，但组合使用可改善。
- 适合关注模型可信度与人类对齐的研究者参考。
近期研究试图量化大语言模型的不确定性，以增强模型控制并调节用户信任。以往工作聚焦于理论基础扎实或反映模型平均行为的不确定性度量。本文考察多种不确定性度量，旨在识别与人类群体层面不确定性相关的指标。结果发现，贝叶斯类度量及一种变体熵度量（top-k entropy）随模型规模变化时，与人类行为具有较高一致性。然而，部分强指标在模型规模增大时人类契合度下降。通过多元线性回归分析，组合多个度量可实现与人类对齐的可比效果，且降低对模型规模的依赖。
原文摘要 · Abstract (English)
Recent work has sought to quantify large language model uncertainty to facilitate model control and modulate user trust. Previous works focus on measures of uncertainty that are theoretically grounded or reflect the average overt behavior of the model. In this work, we investigate a variety of uncertainty measures, in order to identify measures that correlate with human group-level uncertainty. We find that Bayesian measures and a variation on entropy measures, top-k entropy, tend to agree with human behavior as a function of model size. We find that some strong measures decrease in human-similarity with model size, but, by multiple linear regression, we find that combining multiple uncertainty measures provide comparable human-alignment with reduced size-dependency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。