用随机噪声预训练，让模型初始自信度更真实。
Pretraining with random noise for uncertainty calibration
- 用随机噪声和标签预训练网络，修正初始过自信问题。
- 预训练后模型对未知数据保持低置信度，识别能力更强。
- 无需额外处理，适合需要可靠置信度的场景。
不确定性校准在机器学习应用中至关重要，但依然面临挑战。许多模型存在幻觉现象——在未充分训练时表现出过高置信度却错误判断。本文发现，深度学习中常见的随机初始化是导致该问题的根本原因。受发育神经科学启发，我们提出仅通过随机噪声和标签进行预训练的方法，有效降低初始过自信现象，使初始置信度趋近随机水平。这确保了后续数据训练过程中置信度与实际准确率对齐，无需额外预处理或后处理。经预校准的网络能更好识别分布外数据，在其上表现低置信度，从而解决置信度失准问题。
原文摘要 · Abstract (English)
Uncertainty calibration is crucial for various machine learning applications, yet it remains challenging. Many models exhibit hallucinations - confident yet inaccurate responses - due to miscalibrated confidence. Here, we show that the common practice of random initialization in deep learning, often considered a standard technique, is an underlying cause of this miscalibration, leading to excessively high confidence in untrained networks. Our method, inspired by developmental neuroscience, addresses this issue by simply pretraining networks with random noise and labels, reducing overconfidence and bringing initial confidence levels closer to chance. This ensures optimal calibration, aligning confidence with accuracy during subsequent data training, without the need for additional pre- or post-processing. Pre-calibrated networks excel at identifying "unknown data," showing low confidence for out-of-distribution inputs, thereby resolving confidence miscalibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。