让大模型预测时自带可信度评估,避免盲目自信。
From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation

- 用强化学习将误差反馈融入训练,同时优化预测和信心信号。
- 在医疗金融任务中,信心估计准确率显著优于传统方法。
- 适合需要可靠预测结果的高风险场景,如医疗诊断、金融分析。
大语言模型可从非结构化输入中进行基于语言的定量预测,但易产生幻觉和过度自信错误,因此不仅要知道预测结果,更要了解其可信程度。我们提出CARE-PPO,一种强化学习框架,将损失预测与不确定性估计相结合,实现语言型定量预测中数值估计与可信度信号的联合学习。CARE-PPO采用基于预测误差的置信度对齐奖励(CARE),为智能体提供密集的误差感知反馈,并引导评判者学习与预测质量对齐的价值函数。推理时,将评判者重用于信心估计。在医疗与金融两个真实任务及两个Qwen-3模型规模(4B和8B)下,CARE-PPO均取得优异的定量预测表现,且通过评判者生成的信心估计显著优于基于对数几率和语言描述的基线方法。这些优势在跨领域、跨语言的真实分布外场景中依然保持。此外,该方法降低了通用指令遵循提示下的任务过拟合,体现出强化学习微调相比监督微调更优的泛化能力。
原文摘要 · Abstract (English)
LLMs can perform language-based quantitative prediction from unstructured inputs, but remain susceptible to hallucinations and overconfident errors, making it critical to know not only what a model predicts, but when its predictions can be trusted. We introduce CARE-PPO, a reinforcement learning framework that establishes a connection between loss prediction for uncertainty estimation and actor-critic PPO fine-tuning, enabling joint learning of accurate numerical estimates and reliable confidence signals in language-based quantitative prediction. CARE-PPO uses a Confidence-Aligned Reward for Estimation, defined as a function of prediction error, to provide dense error-aware feedback to the actor while inducing the critic to learn a value function aligned with prediction quality. During inference, we repurpose the critic as a confidence estimator. Across two real-world tasks in healthcare and finance and two Qwen-3 model scales (4B and 8B), CARE-PPO achieves strong quantitative prediction performance, while producing significantly better-aligned confidence estimates through the critic than logit-based and verbalized baselines. These gains persist under realistic out-of-distribution settings across domains, spanning linguistic and domain shifts. Finally, CARE-PPO reduces task-specific overfitting on general instruction-following prompts, consistent with the broader generalization advantages of RL fine-tuning over supervised approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。