用双重对齐修复微调后大模型的过度自信问题
Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs
- 通过最终分布和推理过程双路径对齐，修正信心漂移与过程漂移
- 仅用一个温度参数就同时解决两类误差，保持微调性能
- 无需标注数据，适合提升各类微调后模型的可靠性
微调可提升大语言模型性能，但常导致信心校准恶化，出现系统性过度自信。现有无监督后处理方法通过将微调模型输出分布匹配到校准良好的预训练模型来缓解此问题，但该静态匹配忽略了微调引入的推理动态变化。我们发现校准误差源于两种机制：(i) 信心漂移——最终置信度上升而中间决策过程基本一致；(ii) 过程漂移——中间推理路径发生偏离。基于此诊断，我们提出 Dual-Align，一种无监督后处理框架，实现双重对齐：通过最终分布匹配纠正信心漂移，通过定位路径偏离层并重校后续推理稳定性来应对过程漂移。该策略仅学习单一温度参数，在不牺牲微调性能的前提下同时修正两类漂移。实验表明，其在多个任务上持续优于基线，显著降低校准误差，接近有监督最优解。
原文摘要 · Abstract (English)
Post-training improves large language models (LLMs) but often worsens confidence calibration, leading to systematic overconfidence. Recent unsupervised post-hoc methods for post-trained LMs (PoLMs) mitigate this by aligning PoLM confidence to that of well-calibrated pre-trained counterparts. However, framing calibration as static output-distribution matching overlooks the inference-time dynamics introduced by post-training. In particular, we show that calibration errors arise from two regimes: (i) confidence drift, where final confidence inflates despite largely consistent intermediate decision processes, and (ii) process drift, where intermediate inference pathways diverge. Guided by this diagnosis, we propose Dual-Align, an unsupervised post-hoc framework for dual alignment in confidence calibration. Dual-Align performs confidence alignment to correct confidence drift via final-distribution matching, and introduces process alignment to address process drift by locating the layer where trajectories diverge and realigning the stability of subsequent inference. This dual strategy learns a single temperature parameter that corrects both drift types without sacrificing post-training performance gains. Experiments show consistent improvements over baselines, reducing calibration errors and approaching a supervised oracle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。