arXiv:2602.22751cs.AI2026-02被引 3

让大模型学会评估自己推理的不确定性,提升真实思考质量。

Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

  • 用令牌概率推导出无额外开销的不确定性代理,实时捕捉模型自信程度。
  • 在数学和问答任务中,推理准确率显著提升,最高达12.3%相对增长。
  • 适合追求可靠推理路径的研究者,尤其适用于需严格逻辑验证的场景。

大型推理模型(LRMs)已成为解决复杂现实任务的强大范式。实践中,这些模型主要通过可验证奖励的强化学习(RLVR)训练,但现有仅基于结果的RLVR流程几乎完全依赖二元正确性信号,严重忽视模型内在不确定性。我们称此为不确定-奖励错配,在此情况下,高与低不确定性解法被同等对待,阻碍了策略从“追求正确答案”向“优化有效推理路径”的转变。这一限制在数学和问答等以推理为核心的任务中尤为关键。为此,我们提出EGPO——一种元认知熵校准框架,将内在不确定性显式引入RLVR以增强LRMs。EGPO利用零开销的熵代理估计样本级不确定性,通过非对称校准机制将之与外在正确性对齐,既保留正确推理,又选择性调控过度自信的失败,实现稳定且具备不确定性感知的策略优化。此外,EGPO无需修改验证器或奖励定义,即可从原本退化的组级采样中恢复有信息量的学习信号。在多个基准上的大量实验表明,该方法带来显著且一致的推理性能提升,为通过元认知熵校准推进大型推理模型提供了原则性路径。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks. In practice, these models are predominantly trained via Reinforcement Learning with Verifiable Rewards (RLVR), yet most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty. We term this discrepancy the uncertainty-reward mismatch, under which high- and low-uncertainty solutions are treated equivalently, preventing the policy from "Know What You Know" and impeding the shift from optimizing for correct answers to optimizing effective reasoning paths. This limitation is especially critical in reasoning-centric tasks such as mathematics and question answering, where performance hinges on the quality of the model's internal reasoning process rather than mere memorization of final answers. To address this, we propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs. EGPO estimates per-sample uncertainty using a zero-overhead entropy proxy derived from token-level likelihoods and aligns it with extrinsic correctness through an asymmetric calibration mechanism that preserves correct reasoning while selectively regulating overconfident failures, thereby enabling stable and uncertainty-aware policy optimization. Moreover, EGPO recovers informative learning signals from otherwise degenerate group-based rollouts without modifying the verifier or reward definition. Extensive experiments across multiple benchmarks demonstrate that the proposed EGPO leads to substantial and consistent improvements in reasoning performance, establishing a principled path for advancing LRMs through metacognitive entropy calibration.

强化学习推理模型不确定性建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。