用信心感知正则化，让大模型推理更准更可信。
CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models
- 用偏斜反KL散度替代传统正则，根据信心程度动态调整惩罚
- 在保持基线校准性的同时，推理能力接近无约束RFT水平
- 适合追求高可靠推理的模型部署场景
强化微调（RFT）已成为激发大语言模型推理能力的有效范式。然而我们发现关键矛盾：无约束RFT虽提升推理表现，却显著加剧幻觉并恶化模型校准性；而基于反KL（RKL）约束的RFT虽保留可信度，但因对探索性偏离施加无界惩罚，限制了推理提升。为此，我们提出CARE-RFT（信心锚定正则化强化微调），以偏斜反KL散度替代标准反KL正则。CARE-RFT提供信心敏感的惩罚机制：对有把握且一致获奖励的探索施加有界惩罚以促进推理，其他情况则无界惩罚以维持校准性。在多个模型规模和RFT算法上的实验表明，CARE-RFT实现了卓越平衡——推理性能媲美无约束RFT，同时恢复了基线模型的可信度与校准性。本工作证明，精心设计的信心感知正则化是构建兼具能力与可信度推理模型的关键。
原文摘要 · Abstract (English)
Reinforcement finetuning (RFT) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models. However, we identify a critical trade-off: while unconstrained RFT achieves strong reasoning performance, it severely compromises model trustworthiness by amplifying hallucination and worsening calibration; conversely, RKL-constrained RFT preserves trustworthiness but limits reasoning gains due to its unbounded penalty on exploratory deviations. To resolve this tension, we introduce CARE-RFT (Confidence-Anchored Regularized Reinforcement Finetuning), a novel method that replaces standard reverse KL regularization with a skew reverse KL divergence. CARE-RFT provides a confidence-sensitive penalty: it is bounded for confident, consistently rewarded explorations to enable reasoning, while unbounded elsewhere to preserve calibration. Extensive experiments across multiple model scales and RFT algorithms show that CARE-RFT achieves a superior balance, matching the reasoning performance of unconstrained RFT while recovering the trustworthiness and calibration of the base model. Our work establishes that careful, confidence-aware regularization is key to building both capable and trustworthy reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。