通过对比推理过程提升大模型自我认知能力，让回答更可信。
Enhancing LLM Metacognition via Cognitive Pairwise Training
- 用推理路径对比训练模型识别可靠与错误推理
- 14B模型上数学平均分提升2.2点，拒答准确率提升5.6点
- 适合关注模型可靠性与拒绝策略优化的研究者
基于可验证奖励的强化学习（RLVR）已成为大语言模型推理的核心方法，但其结果级奖励可能导致模型在证据不足时仍给出自信回答。现有SFT或强化学习方法多在响应层面教模型拒绝或表达不确定性，易导致回避行为过拟合，而非真正提升推理可靠性。为此，我们提出认知成对训练（CPT），一种认知中段对齐阶段，将推理轨迹的成对比较转化为可复用的对齐信号。通过学习区分可信与有缺陷的推理，CPT促使模型内化推理质量判别边界，而非记忆表面拒绝模式。在五个模型规模和三个模型家族中，CPT均提升了推理与元认知之间的权衡。在14B模型上，CPT+RL相比标准SFT+RL基准提升2.2分数学平均分和5.6分拒答F1值。进一步分析表明，CPT提升了推理轨迹质量，并在评估与训练设置中表现出强鲁棒性与可扩展性。代码与模型已开源：https://github.com/Tsinghua-dhy/CPT。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable. Existing SFT or RL methods mainly teach LLMs to refuse or express uncertainty at the response level, which can overfit abstention behavior rather than improve reasoning reliability. To address this limitation, we propose Cognitive Pairwise Training (CPT), a cognitive mid-training alignment stage that turns pairwise comparisons over reasoning traces into a reusable alignment signal. By learning to distinguish trustworthy from flawed reasoning, CPT encourages the model to internalize a reasoning-quality discrimination boundary rather than memorize surface refusal patterns. Across five model scales and three model families, CPT improves the reasoning--metacognition trade-off. At 14B, CPT+RL outperforms the standard SFT+RL pipeline by +2.2 math-average points and +5.6 abstention-F1 points. Further analyses show that CPT improves trace quality and exhibits strong robustness and scalability across evaluation and training settings. Code and models are released at https://github.com/Tsinghua-dhy/CPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。