通过对比学习缓解大模型自蒸馏中的暴露偏差问题
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
- 将自蒸馏重构为对比学习，用预测熵区分正负样本
- 在13个复杂推理任务中显著提升长序列交互的性能
- 适合需要稳定训练和强泛化能力的复杂对话系统
近期大型语言模型的后训练越来越依赖可验证奖励的强化学习（RLVR）或基于策略的自蒸馏（OPSD）。尽管OPSD提供密集的逐层监督，但因其教师模型拥有特权信息，天然存在暴露偏差。在多轮代理设置中，这会导致推理路径收敛，失去清晰优化方向。为此，我们提出对比强化策略优化（CRPO），将代理型OPSD重新建模为对比学习框架。通过预测熵区分正样本位置（反映探索性思考）与负样本位置（暴露偏差），CRPO进行分组对比，以保留可靠且细粒度的优化信号。在13个具有挑战性的推理与深度搜索基准上的广泛评估表明，CRPO持续优于现有强化学习与自蒸馏基线，在长时程交互中显著提升训练稳定性和泛化能力。
原文摘要 · Abstract (English)
Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。