提出自适应温度缩放，让大模型输出更可信。
Calibrating Language Models with Adaptive Temperature Scaling
- 按每个词预测动态调整温度参数，实现精细化校准。
- 在三个评测集上校准度提升10%-50%，且不影响RLHF性能。
- 适合关注模型置信度可靠性、需高可信生成的场景。
大型语言模型的有效性不仅体现在生成准确内容的能力，还体现在其置信度得分与输出正确概率的匹配程度——即校准度。尽管无监督预训练可使模型具备良好的条件概率校准能力，但近期研究发现，经过人类反馈强化学习（RLHF）微调后，模型的校准能力显著下降。本文提出自适应温度缩放（ATS），一种后处理校准方法，能够为每次词级预测动态预测温度缩放参数。该参数基于词级别特征自适应调整，并在标准监督微调（SFT）数据集上拟合。其自适应特性有效应对了RLHF微调后出现的多样化校准偏差。实验表明，相较于现有校准方法，ATS在三个下游自然语言评测基准上提升了超过10%-50%的校准效果，且不损害来自RLHF的性能提升。
原文摘要 · Abstract (English)
The effectiveness of large language models (LLMs) is not only measured by their ability to generate accurate outputs but also by their calibration-how well their confidence scores reflect the probability of their outputs being correct. While unsupervised pre-training has been shown to yield LLMs with well-calibrated conditional probabilities, recent studies have shown that after fine-tuning with reinforcement learning from human feedback (RLHF), the calibration of these models degrades significantly. In this work, we introduce Adaptive Temperature Scaling (ATS), a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction. The predicted temperature values adapt based on token-level features and are fit over a standard supervised fine-tuning (SFT) dataset. The adaptive nature of ATS addresses the varying degrees of calibration shift that can occur after RLHF fine-tuning. ATS improves calibration by over 10-50% across three downstream natural language evaluation benchmarks compared to prior calibration methods and does not impede performance improvements from RLHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。