arXiv:2605.24396cs.AI2026-05被引 4

通过抑制过早自信,提升大模型推理准确率与透明度。

Understanding and Mitigating Premature Confidence for Better LLM Reasoning

  • 用渐进式信心调节训练模型,避免过早锁定答案
  • 在数学和科学任务上准确率最高提升42个百分点
  • 适合追求可靠推理的大模型应用开发者

当前语言模型的长链思维常存在逻辑漏洞和无根据跳跃,限制了测试时计算资源的增益。直接提升推理质量需过程奖励模型,但逐步标注成本高且稀缺。我们发现模型在推理过程中信心的演变可作为信号:过早自信(early confidence commitment)强烈预测推理错误,且跨任务、跨模型规模均显著。为此提出渐进式信心塑造方法,基于强化学习训练模型随推理逐步更新信心,而非提前锁定答案——无需外部标签或奖励模型。该方法在1.5B至8B参数规模下均有效:在Countdown任务中准确率提升3.2倍(+42.0pp),错误推理下降48pp;在AIME任务中Pass@64提升6.6pp。同时增强模型可信度:在安全基准上更透明地暴露误导性内容。控制实验表明,过早自信随模型规模和任务难度上升,其缓解效果也同步增强。

原文摘要 · Abstract (English)

Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps, limiting the gains from additional test-time compute. Improving reasoning quality directly would require process reward models, but the step-level annotations needed to train them are expensive and scarce. We find such a signal in how the model's confidence evolves during reasoning: premature confidence, the tendency to commit to an answer early and use the remaining tokens to rationalize it, strongly predicts flawed reasoning across tasks and model scales. We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early -- rewarding gradual confidence growth and penalizing early commitment, with no external labels or reward models. The method improves accuracy and reasoning quality from 1.5B to 8B parameters across arithmetic (Countdown), math (DAPO, AIME), and science (ScienceQA): on Countdown, accuracy improves 3.2x (+42.0pp) and flawed reasoning drops 48pp; on AIME, Pass@64 improves 6.6pp. Consistent with this mechanism, the method also improves faithfulness: on a safety benchmark, our models more transparently surface misleading content in their reasoning traces rather than concealing it. Controlled experiments reveal that the problem and its remedy scale together: premature confidence grows with model size and task difficulty, and so do the gains from addressing it.

大模型推理信心建模链式思考可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。