用对比分歧提升强化学习中的正确性判断，让模型更懂对错。
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
- 通过对比参考生成与普通生成的分歧，感知每个词的正确性。
- 在领域内和跨领域任务上显著优于基于熵的方法。
- 适合需要高准确率且注重推理正确性的生成任务。
带有可验证奖励的强化学习（RLVR）通常使用熵进行优势塑造，但熵无法区分有益的不确定性与有害的混淆，限制了其作为正确性信号的效果。我们提出对比策略优化（CPO），利用参考引导生成分布与普通生成分布之间的词级对比分歧，实现面向正确性的优势塑造。理论与实证结果表明，该分歧能可靠反映词级正确性。我们进一步证明，在线策略蒸馏是CPO的一个特例，其中后验分布由外部教师模型提供。CPO还解决了零优势问题。在域内和域外基准测试中，CPO显著优于基于熵的RLVR方法，同时保持强泛化能力。进一步分析显示，正确与错误响应分别自然支持探索与利用，平衡两者可获得最佳性能。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。