对比人眼对不确定性的感知与神经网络的预测，发现两者差距大。
Uncertainty Estimation by Human Perception versus Neural Models
- 用人类标注的分歧与信心数据，比较模型与人的不确定性判断。
- 现有模型不确定性估计与人类感知相关性弱，任务间差异明显。
- 引入人类软标签可提升模型校准度，不牺牲准确率，适合可信AI研究者。
现代神经网络虽预测准确率高，但校准不足，常在错误时仍过度自信。本文通过三个视觉基准数据集（含人类分歧与众包置信度标注），评估模型预测不确定性与人类感知不确定性的相关性。结果表明，当前方法与人类直觉仅存在微弱关联，相关性在不同任务和不确定性度量间差异显著。值得注意的是，将人类提供的软标签融入训练过程，可在不损失准确率的前提下改善模型校准。研究揭示了模型与人类在不确定性认知间的持续差距，并表明利用人类洞察有助于构建更可信的AI系统。
原文摘要 · Abstract (English)
Modern neural networks (NNs) often achieve high predictive accuracy but are poorly calibrated, producing overconfident predictions even when wrong. This miscalibration poses serious challenges in applications where reliable uncertainty estimates are critical. In this work, we investigate how human perceptual uncertainty compares to uncertainty estimated by NNs. Using three vision benchmarks annotated with both human disagreement and crowdsourced confidence, we assess the correlation between model-predicted uncertainty and human-perceived uncertainty. Our results show that current methods only weakly align with human intuition, with correlations varying significantly across tasks and uncertainty metrics. Notably, we find that incorporating human-derived soft labels into the training process can improve calibration without compromising accuracy. These findings reveal a persistent gap between model and human uncertainty and highlight the potential of leveraging human insights to guide the development of more trustworthy AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。