arXiv:2506.07295cs.CL2025-06被引 80

温度影响大模型表现,不同任务需调不同温。

Exploring the Impact of Temperature on Large Language Models:Hot or Cold?

  • 用BERT模型根据提示自动选最佳采样温度。
  • 小中模型在SuperGLUE上性能提升明显。
  • 模型越大,温度突变点越高,适合量化部署者参考。

采样温度是大语言模型中的关键超参数,通过调整softmax前的logits来改变输出词元分布。本研究系统评估了0到2范围内温度对六项能力测试集的影响,使用三种规模的开源模型(小:1B–4B,中:6B–13B,大:40B–80B)。结果表明温度对不同技能有特定影响,凸显实际应用中最优温度选择的复杂性。为此我们提出基于BERT的温度选择器,利用观测到的效果为给定提示选取最优温度,在SuperGLUE数据集上显著提升了小中模型表现。研究还扩展至FP16精度推理,发现温度效应与4位量化模型一致。在三个量化模型中将温度测试至4.0,发现随模型增大,性能突变点(Mutation Temperature)上升。

原文摘要 · Abstract (English)

The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output tokens. Recent studies have challenged the Stochastic Parrots analogy by demonstrating that LLMs are capable of understanding semantics rather than merely memorizing data and that randomness, modulated by sampling temperature, plays a crucial role in model inference. In this study, we systematically evaluated the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities, conducting statistical analyses on open source models of three different sizes: small (1B--4B), medium (6B--13B), and large (40B--80B). Our findings reveal distinct skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection in practical applications. To address this challenge, we propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt. We demonstrate that this approach can significantly improve the performance of small and medium models in the SuperGLUE datasets. Furthermore, our study extends to FP16 precision inference, revealing that temperature effects are consistent with those observed in 4-bit quantized models. By evaluating temperature effects up to 4.0 in three quantized models, we find that the Mutation Temperature -- the point at which significant performance changes occur -- increases with model size.

大模型温度调节推理优化量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。