arXiv:2506.09338stat.MLcs.AI2025-06NeurIPS被引 19

校准推理模型的不确定性,让大模型更聪明地分配计算资源。

Know What You Don't Know: Uncertainty Calibration of Process Reward Models

  • 用分位数回归校准推理模型输出的置信度
  • 校准后可减少40%以上推理开销,且准确率不变
  • 适合需要高效推理的数学推理场景

过程奖励模型(PRMs)在指导大语言模型推理阶段扩展算法中起核心作用。然而我们发现,即使是最先进的PRMs也存在校准不良问题,尤其当使用较小的语言模型完成推理轨迹时,倾向于高估部分推理步骤导致正确答案的概率。为此,我们提出一种基于分位数回归的校准方法,使PRM输出更贴近真实成功概率。利用这些校准后的成功率估计及其置信区间,我们引入一种实例自适应扩展(IAS)框架,根据每个推理步骤的成功可能性动态调整计算预算。与传统固定轨迹数的方法不同,该方法针对每条查询和每个步骤进行自适应调节。在数学推理基准上的实验表明:(i) 本方法实现小校准误差,优于基线;(ii) 校准是有效IAS的关键;(iii) IAS策略在保持最终答案准确率的同时,显著降低推理成本,对更确定的问题使用更少计算量。

原文摘要 · Abstract (English)

Process reward models (PRMs) play a central role in guiding inference-time scaling algorithms for large language models (LLMs). However, we observe that even state-of-the-art PRMs can be poorly calibrated. Specifically, they tend to overestimate the success probability that a partial reasoning step will lead to a correct final answer, particularly when smaller LLMs are used to complete the reasoning trajectory. To address this, we present a calibration approach -- performed via quantile regression -- that adjusts PRM outputs to better align with true success probabilities. Leveraging these calibrated success estimates and their associated confidence bounds, we introduce an \emph{instance-adaptive scaling} (IAS) framework that dynamically adjusts the compute budget based on the estimated likelihood that a partial reasoning trajectory will yield a correct final answer. Unlike conventional methods that allocate a fixed number of reasoning trajectories per query, this approach adapts to each instance and reasoning step when using our calibrated PRMs. Experiments on mathematical reasoning benchmarks show that (i) our PRM calibration method achieves small calibration error, outperforming the baseline methods, (ii) calibration is crucial for enabling effective IAS, and (iii) the proposed IAS strategy reduces inference costs while maintaining final answer accuracy, utilizing less compute on more confident problems as desired.

推理优化不确定性校准大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。