通过对比正确与错误答案,精准识别推理中的关键步骤。
CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

- 用正确与错误答案对比,判断每个词是否为真正推理步骤。
- 在20亿和40亿参数下,准确率分别达43.43%和60.56%。
- 无需额外采样,适合数学推理等需要精确奖励分配的任务。
当模型在可验证奖励的强化学习(RLVR)中生成正确解时,所有词元均获得相同奖励信号,无论其是决定性推理步骤还是语法填充。现有方法通过将正确答案作为教师指导模型,但可能因答案泄露污染训练,或产生无法区分关键步骤与填充物的弱信号。本文提出对比证据策略优化(CEPO),在每个词元处提出更精准的问题:不仅问‘正确答案是否支持该词元’,还问‘错误答案是否排斥该词元’。同时满足两者者为真实推理步骤,均不满足者为填充。错误答案教师由训练批次内已拒绝的轨迹构建,无额外采样成本。理论证明CEPO继承先前最优方法的所有结构安全性,且严格增强关键词元的信用分配,而在填充位置提升归零。实验表明,在五个多模态数学推理基准上,CEPO在2B和4B规模下平均准确率分别为43.43%和60.56%,优于相同预算下的GRPO(41.17%和57.43%)。分布匹配自蒸馏方法(OPSD、SDPO)表现低于未训练基线,实证验证了理论预测的信息泄露问题。代码已开源。
原文摘要 · Abstract (English)
When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler. A natural fix is to condition the model on the correct answer as a teacher, identifying tokens it would have generated differently had it known the answer. Prior work shows this either corrupts training by leaking the answer into the gradient, or produces a weak signal that cannot distinguish decisive steps from filler, since both look equally surprising relative to the model's baseline. We propose Contrastive Evidence Policy Optimization (CEPO), which asks a sharper question at every token: not just "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?" A token satisfying both is a genuine reasoning step; one satisfying neither is filler. The wrong-answer teacher is constructed from rejected rollouts already in the training batch, incurring no additional sampling cost. We prove CEPO inherits all structural safety guarantees of the prior state of the art while strictly sharpening credit at decisive tokens, with the improvement vanishing exactly at filler positions. Empirically, CEPO achieves 43.43% and 60.56% average accuracy across five multimodal mathematical reasoning benchmarks at 2B and 4B scale, respectively, versus 41.17% and 57.43% for GRPO under identical training budgets. Distribution-matching self-distillation methods (OPSD, SDPO) fall below the untrained baseline, empirically confirming the information leakage our theory predicts. Our code is available at https://github.com/ahmedheakl/CEPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。