大模型微调后信心常不准，因先验知识导致过自信。
Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?
- 根据模型先验知识差异，动态调整微调策略
- 在7个任务上使校准误差降低57%（Llama3-8B）
- 适合追求可信度的高风险人机交互场景
微调后的大型语言模型常出现校准不良，其置信度与实际表现不符。尽管从零训练模型的校准问题已有广泛研究，但大模型先验知识在微调过程中的影响仍被忽视。我们的研究发现，真实世界微调数据普遍包含与模型先验知识重叠的内容，这会引发过自信现象；而引入新知识则有助于改善校准。这种矛盾在于：大模型丰富的先验知识虽增强任务泛化能力，却因不可避免的知识重叠损害了校准性能。为此，我们提出CogCalib——一种基于认知感知的框架，依据模型先验知识差异采用针对性学习策略。在3个大模型家族、7个任务上的实验表明，CogCalib显著提升校准效果，相比标准微调，平均降低57%的ECE（Llama3-8B），且在跨域任务中表现良好，增强了领域特定大模型的客观性与可靠性，适用于关键人机交互应用。
原文摘要 · Abstract (English)
Fine-tuned Large Language Models (LLMs) often demonstrate poor calibration, with their confidence scores misaligned with actual performance. While calibration has been extensively studied in models trained from scratch, the impact of LLMs' prior knowledge on calibration during fine-tuning remains understudied. Our research reveals that LLMs' prior knowledge causes potential poor calibration due to the ubiquitous presence of known data in real-world fine-tuning, which appears harmful for calibration. Specifically, data aligned with LLMs' prior knowledge would induce overconfidence, while new knowledge improves calibration. Our findings expose a tension: LLMs' encyclopedic knowledge, while enabling task versatility, undermines calibration through unavoidable knowledge overlaps. To address this, we propose CogCalib, a cognition-aware framework that applies targeted learning strategies according to the model's prior knowledge. Experiments across 7 tasks using 3 LLM families prove that CogCalib significantly improves calibration while maintaining performance, achieving an average 57\% reduction in ECE compared to standard fine-tuning in Llama3-8B. These improvements generalize well to out-of-domain tasks, enhancing the objectivity and reliability of domain-specific LLMs, and making them more trustworthy for critical human-AI interaction applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。