让大模型的预测信心更靠谱，不瞎自信也不乱怀疑。
CATTO: Balancing Preferences and Confidence in Language Models
- 用新目标函数让模型信心与实际正确率对齐
- 校准后错误率下降，答题准确率不变甚至略升
- 适合需要可靠置信度的应用场景
大语言模型常能准确预测下一个词，但其预测信心往往失准：高信心时易出错，低信心时反而可能正确。偏好对齐方法进一步破坏了预测概率与正确性之间的联系。我们提出校准感知的逐标记训练目标（CATTO），使模型预测信心与实际正确率对齐，可与原有偏好优化目标结合使用。实验表明，相比直接偏好优化（DPO），CATTO在分布内将期望校准误差（ECE）降低2.22%–7.61%，分布外降低1.46%–10.44%；相比最强的DPO基线，分布内降低0.22%–1.24%，分布外降低1.23%–5.07%。该改进未牺牲任务准确性，在五个数据集上多选题问答准确率保持或小幅提升。我们还引入Confidence@k，一种基于校准概率的测试时缩放机制，实现贝叶斯最优输出词选择。
原文摘要 · Abstract (English)
Large language models (LLMs) often make accurate next token predictions but their confidence in these predictions can be poorly calibrated: high-confidence predictions are frequently wrong, and low-confidence predictions may be correct. This miscalibration is exacerbated by preference-based alignment methods breaking the link between predictive probability and correctness. We introduce a Calibration Aware Token-level Training Objective (CATTO), a calibration-aware objective that aligns predicted confidence with empirical prediction correctness, which can be combined with the original preference optimization objectives. Empirically, CATTO reduces Expected Calibration Error (ECE) by 2.22%-7.61% in-distribution and 1.46%-10.44% out-of-distribution compared to direct preference optimization (DPO), and by 0.22%-1.24% in-distribution and 1.23%-5.07% out-of-distribution compared to the strongest DPO baseline. This improvement in confidence does not come at a cost of losing task accuracy, where CATTO maintains or slightly improves multiple-choice question-answering accuracy on five datasets. We also introduce Confidence@k, a test-time scaling mechanism leveraging calibrated token probabilities for Bayes-optimal selection of output tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。