arXiv:2502.18389cs.CL2025-02被引 15

提出蒙特卡洛温度法,无需调参即可稳定量化大模型不确定性

Monte Carlo Temperature: a robust sampling strategy for LLM's uncertainty quantification methods

  • 采用随机温度采样策略替代固定温度,避免繁琐调参
  • 在多种温度下保持稳定不确定性估计,性能优于传统方法
  • 效果媲美最优温度设置,显著降低计算开销,适合实际部署

大型语言模型(LLM)的不确定性量化(UQ)对安全可靠部署至关重要,尤其在错误输出可能带来严重后果的关键场景中。现有UQ方法通常通过非零温度采样多次查询模型,生成多样化输出以估算不确定性,但温度参数的选择影响尚未被充分研究。我们的分析表明,温度对不确定性估计质量具有根本性影响。传统优化最佳温度需进行昂贵的超参数优化(HPO),且每次新模型-数据组合都需重复。为此,我们提出蒙特卡洛温度(MCT)策略,消除温度校准需求。分析显示:1)MCT在广泛温度范围内提供更鲁棒的不确定性估计;2)相比依赖HPO的固定温度策略,MCT提升UQ性能;3)其效果达到与理想‘黄金温度’相当的统计水平,后者需代价高昂的HPO实现。结果表明,有效UQ可无需温度校准的计算负担。

原文摘要 · Abstract (English)

Uncertainty quantification (UQ) in Large Language Models (LLMs) is essential for their safe and reliable deployment, particularly in critical applications where incorrect outputs can have serious consequences. Current UQ methods typically rely on querying the model multiple times using non-zero temperature sampling to generate diverse outputs for uncertainty estimation. However, the impact of selecting a given temperature parameter is understudied, and our analysis reveals that temperature plays a fundamental role in the quality of uncertainty estimates. The conventional approach of identifying optimal temperature values requires expensive hyperparameter optimization (HPO) that must be repeated for each new model-dataset combination. We propose Monte Carlo Temperature (MCT), a robust sampling strategy that eliminates the need for temperature calibration. Our analysis reveals that: 1) MCT provides more robust uncertainty estimates across a wide range of temperatures, 2) MCT improves the performance of UQ methods by replacing fixed-temperature strategies that do not rely on HPO, and 3) MCT achieves statistical parity with oracle temperatures, which represent the ideal outcome of a well-tuned but computationally expensive HPO process. These findings demonstrate that effective UQ can be achieved without the computational burden of temperature parameter calibration.

不确定性量化大模型采样策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。