用对比学习提升大模型推理的鲁棒性，防止幻觉
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
- 在强化学习中引入对比损失，让模型学习正确推理路径的共性结构
- 在多个推理基准上显著提升基线模型的泛化与鲁棒性
- 适合关注大模型推理质量与防幻觉的研究者
基于可验证奖励的强化学习（RLVR）显著提升了大语言模型（LLMs）的推理能力。然而，RLVR仅依赖最终答案作为奖励信号，忽略了中间推理步骤的正确性。在过程错误但结果正确的轨迹上训练，会导致幻觉和答案复制，严重损害模型的泛化与鲁棒性。为此，本文提出将对比学习引入策略优化（CLIPO），通过在成功轨迹上优化对比损失，引导模型捕捉不同正确推理路径间的不变结构。相比原RLVR中单路径监督，该方法提供了更强的跨轨迹正则化，有效缓解了步骤级推理不一致问题并抑制幻觉生成。实验表明，CLIPO在多个推理基准上持续优于多种RLVR基线，显著提升了策略优化的泛化与鲁棒性。代码与训练方案见https://github.com/Qwen-Applications/CLIPO。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。