温度缩放能调控模型不确定性，但对大语言模型的多样性影响被高估。
The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling
- 通过信息投影几何解释温度缩放的本质
- 升温会提高分类器熵，但未必提升语言模型多样性
- 是唯一不改变硬预测结果的线性缩放方法
温度缩放是一种简单有效的概率模型不确定性调控方法，广泛用于改善分类器校准和调节大语言模型（LLMs）的随机性。尽管应用广泛，其理论性质仍缺乏严谨分析。本文揭示：在分类任务中，升高温度普遍增加模型不确定性（即熵）；但在LLMs中，挑战了“升温提升多样性”的常见观点。研究提出两个新刻画：一是温度缩放等价于原模型在固定熵约束下的信息投影；二是它作为更通用的线性缩放器（如矩阵缩放、狄利克雷校准）的子集，是唯一不改变模型硬预测结果的方法。
原文摘要 · Abstract (English)
Temperature scaling is a simple method that allows to control the uncertainty of probabilistic models. It is mostly used in two contexts: improving the calibration of classifiers and tuning the stochasticity of large language models (LLMs). In both cases, temperature scaling is the most popular method for the job. Despite its popularity, a rigorous theoretical analysis of the properties of temperature scaling has remained elusive. We investigate here some of these properties. For classification, we show that increasing the temperature increases the uncertainty in the model in a very general sense (and in particular increases its entropy). However, for LLMs, we challenge the common claim that increasing temperature increases diversity. Furthermore, we introduce two new characterisations of temperature scaling. The first one is geometric: the tempered model is shown to be the information projection of the original model onto the set of models with a given entropy. The second characterisation clarifies the role of temperature scaling as a submodel of more general linear scalers such as matrix scaling and Dirichlet calibration: we show that temperature scaling is the only linear scaler that does not change the hard predictions of the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。