用模型自信度增强奖励信号，让大模型强化学习更高效
ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
- 结合模型自信心与可验证结果生成细粒度奖励
- 在多个数据集上提升性能并减少推理耗 token
- 可作为插件适配现有强化学习方法
强化学习已成为超越预训练和指令微调的大型语言模型优化标准范式。主流方法为可验证奖励强化学习（RLVR），利用自动可验证结果（如正确性或可执行性）生成奖励信号。尽管高效，该框架存在两大局限：一是二值反馈过于稀疏，难以捕捉推理过程质量；二是粗粒度奖励可能导致梯度消失。受人类学习机制启发，本文提出一种融合可验证结果与模型自身置信度的强化学习技术。该设计丰富了奖励信号，提供更精细反馈并隐式监督推理过程。实验表明，该方法在多个数据集上提升强化学习性能，降低推理阶段的令牌消耗，且训练成本几乎不变。此外，可作为即插即用模块提升现有先进强化学习方法。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs) beyond pre-training and instruction tuning. A prominent line of work is RL with verifiable rewards (RLVR), which leverages automatically verifiable outcomes (e.g., correctness or executability) to generate reward signals. While efficient, this framework faces two key limitations: First, its binary feedback is too sparse to capture the quality of the reasoning process. Second, its coarse-grained rewards potentially lead to vanishing gradients. Inspired by observations from human learning, we introduce a RL technique that integrates verifiable outcomes with the model's own confidence estimates. This joint design enriches the reward signal, providing finer-grained feedback and implicitly supervising the reasoning process. Experimental results demonstrate that our proposed method enhances RL performance across multiple datasets and reduces token consumption during inference, while incurring negligible additional training cost. Moreover, it can be used as a plug-in module to enhance other state-of-the-art RL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。