用置信度筛选高难度样本,实现高效精准的奖励建模。
CAMEL: Confidence-Gated Reflection for Reward Modeling
- 先快速判断置信度,低置信样本才触发深度反思
- 在三个基准上达82.9%准确率,比之前最优模型高3.2%
- 仅用140亿参数超越700亿参数模型,效率更优
奖励模型在对齐大语言模型与人类偏好方面起着基础性作用。现有方法主要分为两类:标量判别型偏好模型,高效但缺乏可解释性;生成式评判模型,能提供更丰富的推理过程,但计算开销大。我们发现,裁决标记的对数概率差与预测正确性高度相关,可作为无需额外计算成本的实例难度代理指标。基于此,我们提出CAMEL——一种置信度门控的反思框架:先进行轻量级单标记偏好决策,仅对低置信度样本调用反思机制。为促进有效自我修正,我们通过反事实前缀增强进行强化学习训练,使模型接触多样初始裁决,鼓励真实修订。实验表明,CAMEL在三个广泛使用的奖励模型基准上达到82.9%的平均准确率,优于最佳已有模型3.2%,且仅用140亿参数便超越700亿参数模型,建立了更优的准确率-效率帕累托前沿。
原文摘要 · Abstract (English)
Reward models play a fundamental role in aligning large language models with human preferences. Existing methods predominantly follow two paradigms: scalar discriminative preference models, which are efficient but lack interpretability, and generative judging models, which offer richer reasoning at the cost of higher computational overhead. We observe that the log-probability margin between verdict tokens strongly correlates with prediction correctness, providing a reliable proxy for instance difficulty without additional inference cost. Building on this insight, we propose CAMEL, a confidence-gated reflection framework that performs a lightweight single-token preference decision first and selectively invokes reflection only for low-confidence instances. To induce effective self-correction, we train the model via reinforcement learning with counterfactual prefix augmentation, which exposes the model to diverse initial verdicts and encourages genuine revision. Empirically, CAMEL achieves state-of-the-art performance on three widely used reward-model benchmarks with 82.9% average accuracy, surpassing the best prior model by 3.2% and outperforming 70B-parameter models using only 14B parameters, while establishing a strictly better accuracy-efficiency Pareto frontier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。