通过双层优化提升大模型校准能力，改善跨领域泛化表现。
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
- 用预测分布熵最大化作为校准目标，抑制过度自信
- 在多项选择与开放生成任务中实现更好校准效果
- 适合需要可靠置信度的大模型应用，如医疗、金融
偏好对齐常导致大语言模型过度自信且校准不佳。传统后处理温度缩放具有固有的域依赖性：在一个域上拟合的温度参数无法跨域泛化。为此，我们提出在训练过程中调整模型参数以改进校准。通过最大化预测分布的熵作为校准目标，直接抑制过度集中的预测结果。受温度缩放启发，我们采用双层优化框架：下层在参数化损失下训练模型，上层选择损失超参数以最大化熵。为适应大模型规模，我们采用高效的的一阶近似，避免显式二阶计算。在多项选择和开放式生成问答任务中，实验表明该方法能生成校准良好的大模型，尤其在跨域泛化方面表现优异。
原文摘要 · Abstract (English)
Preference alignment often makes large language models (LLMs) overconfident and poorly calibrated. Traditional post-hoc temperature scaling is inherently domain-dependent: a temperature fitted on one domain does not generalize across domains. This motivates us to modify model parameters during training to improve calibration. We propose maximizing the entropy of predictive distributions as the calibration objective, which directly targets overconfidence by discouraging overly concentrated predictions. Inspired by temperature scaling, we realize this through a bilevel optimization formulation, where the lower level trains the model under a parametric loss and the upper level selects loss hyperparameters to maximize entropy. To make the framework practical at LLM scale, we adopt an efficient first-order approximation that avoids explicit second-order computation. Across both multiple-choice and open-ended generative question answering, experiments demonstrate that our method yields well-calibrated LLMs with particular advantages in out-of-domain generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。