分离推理与置信度，解决大模型强化学习中的过度自信问题
Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
- 将推理与校准目标解耦，避免优化冲突
- 在保持准确率的同时显著改善置信度校准
- 适合需要可靠决策的大模型应用
从可验证奖励中进行强化学习（RLVR）显著提升了大语言模型（LLMs）的推理能力，但严重面临校准退化问题，即模型对错误答案表现出过度自信。以往研究尝试直接将校准目标融入优化目标，但我们的理论分析表明，最大化策略准确率与最小化校准误差之间存在根本性的梯度冲突。基于此，我们提出DCPO框架，系统性地解耦推理与校准目标。大量实验表明，DCPO在保持与GRPO相当的准确率的同时，实现了最佳校准性能，并大幅缓解了过度自信问题。本研究为更可靠的LLM部署提供了重要洞察与实用解决方案。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。