让大模型在准确决策的同时保持可信置信度。
Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning
- 设计新强化学习框架，让模型直接调整决策概率以提升校准性。
- 在保持高准确率基础上，将误差校准分数（ECE）降低9点。
- 适合需要可靠置信度的医疗、金融等高风险决策场景。
大型语言模型在决策任务中的应用日益广泛，不仅要求高准确率，还需可靠的置信度估计。良好的置信度使下游系统能判断何时信任模型、何时转用备用机制。本文系统研究了两种主流微调方法——监督微调（SFT）与可验证奖励强化学习（RLVR）中的校准性能。结果表明，尽管RLVR提升了任务表现，却导致模型严重过度自信；而SFT虽性能提升较小，但在分布外情形下仍具有显著更好的校准性。通过实验诊断发现，RLVR中决策标记仅作为推理过程中的提取步骤，不携带置信度信息，阻碍了强化学习发现更校准的决策路径。基于此，我们提出一种校准感知的强化学习方法，直接调整决策标记的概率。该方法在维持RLVR高准确率的同时，显著缓解过度自信问题，使ECE得分最高降低9分。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR's failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR's accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。