发现大模型在深层逐渐修正自信度,而非仅靠最后输出层。
Calibration Across Layers: Understanding Calibration Evolution in LLMs
- 分析模型各层,发现高层存在主动修正信心的阶段。
- 找到残差流中低维校准方向,扰动后提升校准度(ECE/MCE)。
- 适合研究模型可信度、安全性和决策机制的学者。
大型语言模型虽具内在校准能力,但以往研究认为深度神经网络普遍过度自信。近期工作将此现象归因于最终层的特定组件,如熵神经元和未嵌入矩阵的零空间。本文从新视角考察校准如何随网络深度演变。在MMLU基准上分析多个开源模型,我们发现高层存在显著的置信度修正阶段,即在决策确定后仍主动调整信心。此外,我们识别出残差流中一个低维校准方向,其扰动可显著改善校准指标(ECE与MCE),且不影响准确率。结果表明,校准是贯穿前向传播全过程的分布式现象,而不仅限于最终投影层,为理解模型自信调节机制提供了新见解。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated inherent calibration capabilities, where predicted probabilities align well with correctness, despite prior findings that deep neural networks are often overconfident. Recent studies have linked this behavior to specific components in the final layer, such as entropy neurons and the unembedding matrix null space. In this work, we provide a complementary perspective by investigating how calibration evolves throughout the network depth. Analyzing multiple open-weight models on the MMLU benchmark, we uncover a distinct confidence correction phase in the upper/later layers, where model confidence is actively recalibrated after decision certainty has been reached. Furthermore, we identify a low-dimensional calibration direction in the residual stream whose perturbation significantly improves calibration metrics (ECE and MCE) without harming accuracy. Our findings suggest that calibration is a distributed phenomenon, shaped throughout the network forward pass, not just in its final projection, providing new insights into how confidence-regulating mechanisms operate within LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。