通过对比提示优化离散策略,实现更精准的逐标记奖励分配。
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

- 用正负提示对比预测差异,实现逐标记奖励分配
- 在图文生成和思维链任务中显著优于基线方法
- 适合需要精细语义控制的离散策略优化场景
基于组优势的强化学习方法(如GRPO和DAPO)在数学推理和文本到图像生成等任务中表现优异。然而,其依赖样本级奖励导致对所有标记统一赋值,无法捕捉细粒度的标记级贡献。为此,本文提出引导对比策略优化(GCPO),通过对比正负提示下的模型预测,实现逐标记的信用分配。不同于均匀广播样本级优势,GCPO将标记级优势设为对比预测之差,提供更精确的学习信号。实验表明,GCPO在文本到图像生成中聚焦与文本提示对齐的视觉区域,在思维链任务中突出关键关键词。在多个基准测试中,GCPO持续优于GRPO和DAPO,证明其作为通用可扩展离散策略优化方法的有效性。
原文摘要 · Abstract (English)
Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。