研究三种训练方法如何影响大模型推理时的自信度,发现用法需分阶段、看位置。
Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration

- 按推理三阶段评估信心:难易预判、中途停止、结果整合
- OPD在推理前最准,SFT对中途停止最有用,RL在整合阶段最可靠
- 提出位置感知策略PosConf,提升准确率最高达6.1点
大型语言模型通过监督微调(SFT)、强化学习(RL)和在线策略蒸馏(OPD)实现了推理能力的显著提升,但这些方法通常仅以最终答案准确率评估。本文研究它们如何重塑推理过程中的信心。我们提出一个三阶段校准框架,分别评估链式思维生成前、中、后的信心,对应难度估计、早期终止和答案聚合。在数学推理基准上的受控对比显示:OPD提供最有效的推理前信心,SFT给出最强的在线停止信号,而RL产生最可靠的逐步信号用于聚合。进一步发现,信心可靠性与位置相关:RL信心在路径确定后才具信息量,而OPD信心早期有效但后期可能反向校准。基于此,我们提出PosConf——一种基于相对位置的可信信心使用策略。该策略使RL答案聚合比多数投票提升6.1分,且在严格词元预算下持续改进OPD早期停止,最高提升4.3分,避免了其后期反向校准区域。结果表明,推理模型的信心应分阶段、按位置使用。
原文摘要 · Abstract (English)
Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that \emph{confidence in reasoning models should be used both stage-wise and position-awarely}. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。