用基础模型信号无监督校准大模型置信度，解决过自信问题。
BaseCal: Unsupervised Confidence Calibration via Base Model Signals
- 利用基础模型输出作为参考，生成校准后的置信度。
- 在五数据集三模型族上平均降低42.90%的ECE误差。
- 无需标签或模型修改，可直接部署使用。
可靠置信度对信任大模型输出至关重要，但广泛部署的微调大模型（PoLLMs）常因严重过自信而损害可信性。我们发现其对应的基底大模型（base LLMs）通常保持良好校准。由此启发，我们提出两种方法：BaseCal-ReEval通过将PoLLM输出输入基底模型获取平均概率作为置信度；BaseCal-Proj则训练轻量投影层，将PoLLM最终层隐藏状态映射回基底模型状态，再由基底模型输出层生成置信度。该方法为无监督、即插即用方案，无需人工标注或模型修改。在五个数据集和三个大模型家族上的实验表明，相比最优无监督基线，平均降低42.90%的期望校准误差（ECE）。
原文摘要 · Abstract (English)
Reliable confidence is essential for trusting the outputs of LLMs, yet widely deployed post-trained LLMs (PoLLMs) typically compromise this trust with severe overconfidence. In contrast, we observe that their corresponding base LLMs often remain well-calibrated. This naturally motivates us to calibrate PoLLM confidence using the base LLM as a reference. This work proposes two ways to achieve this. A straightforward solution, BaseCal-ReEval, evaluates PoLLM's responses by feeding them into the base LLM to get average probabilities as confidence. While effective, this approach introduces additional inference overhead. To address this, we propose BaseCal-Proj, which trains a lightweight projection to map the final-layer hidden states of PoLLMs back to those of their base LLMs. These projected states are then processed by the base LLM's output layer to derive base-calibrated confidence for PoLLM's responses. Notably, BaseCal is an unsupervised, plug-and-play solution that operates without human labels or LLM modifications. Experiments across five datasets and three LLM families demonstrate the effectiveness of BaseCal, reducing Expected Calibration Error (ECE) by an average of 42.90\% compared to the best unsupervised baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。