不依赖负样本，仅用正向优势样本实现在线强化学习的生成式策略优化。
Positive-Only Drifting Policy Optimization
- 基于漂移模型，仅使用正向优势样本进行策略更新。
- 无需梯度裁剪或信任区域约束，提升训练稳定性。
- 适合追求高效、稳定在线学习的算法研究者与工业应用
在在线强化学习领域，传统高斯策略和基于流的方法常受限于单峰表达能力、复杂的梯度裁剪或严格的信任区域要求。此外，它们均需事后对负样本进行惩罚以修正错误动作。本文提出正向仅漂移策略优化（PODPO），一种无似然且免梯度裁剪的生成式在线强化学习方法。通过漂移模型，PODPO利用优势加权局部对比漂移实现策略更新。仅依赖正向优势样本，即可有效引导动作向高回报区域迁移，并利用生成模型的局部平滑性实现主动纠错。该方法为在线设置下的生成式策略学习开辟了新路径。
原文摘要 · Abstract (English)
In the field of online reinforcement learning (RL), traditional Gaussian policies and flow-based methods are often constrained by their unimodal expressiveness, complex gradient clipping, or stringent trust-region requirements. Moreover, they all rely on post-hoc penalization of negative samples to correct erroneous actions. This paper introduces Positive-Only Drifting Policy Optimization (PODPO), a likelihood-free and gradient-clipping-free generative approach for online RL. By leveraging the drifting model, PODPO performs policy updates via advantage-weighted local contrastive drifting. Relying solely on positive-advantage samples, it elegantly steers actions toward high-return regions while exploiting the inherent local smoothness of the generative model to enable proactive error prevention. In doing so, PODPO opens a promising new pathway for generative policy learning in online settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。