无需奖励函数，用对比学习实现高效的在线强化学习。
Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation
- 基于对比学习直接构建策略优势，融合标准PPO优化。
- 18项任务中14项超越旧有对比强化学习方法，12项媲美带奖励的PPO。
- 适用于连续与离散动作空间，支持单智能体和多智能体场景。
对比强化学习（CRL）通过状态-动作与目标表示的对比目标学习条件化的Q值，无需人工设计奖励函数。尽管在自监督强化学习方面表现优异，现有所有CRL算法均依赖离线策略优化，且主要局限于连续动作空间，对离散环境研究不足。这使得CRL与广泛采用的现代在线策略训练流程脱节。为建立首个连接，本文提出对比近端策略优化（CPPO）。CPPO是一种在线对比强化学习算法，直接从对比Q值导出策略优势，并通过标准PPO目标进行优化，无需奖励函数或经验回放缓冲区。我们在连续与离散、单智能体与协作多智能体任务上评估了CPPO。结果表明，尽管在线策略本身具有优势，但CPPO在18个任务中的14个显著优于先前的CRL基线，且在12个任务中达到或超过使用人工密集奖励的PPO性能。
原文摘要 · Abstract (English)
Contrastive reinforcement learning (CRL) learns goal-conditioned Q-values through a contrastive objective over state-action and goal representations, removing the need for hand-crafted reward functions. Despite impressive success in achieving viable self-supervised learning in RL, all existing CRL algorithms rely on off-policy optimisation and are mostly constrained to continuous action spaces, with little research invested in discrete environments. This leaves CRL disconnected from widely used and effective, modern on-policy training pipelines adopted across both single-agent and multi-agent RL in continuous and discrete environments. To establish a first connection, we introduce Contrastive Proximal Policy Optimisation (CPPO). CPPO is an on-policy contrastive RL algorithm that derives policy advantages directly from contrastive Q-values and optimises them via the standard PPO objective, without requiring a reward function or a replay buffer. We evaluate CPPO across continuous and discrete, single-agent and cooperative multi-agent tasks. Whilst the existence of an on-policy approach is inherently useful, we observe that \textbf{CPPO not only significantly outperforms the previous CRL baselines in 14 out of 18 tasks, but also matches or exceeds PPO's performance, which uses hand-crafted dense rewards, in 12 out of the 18 tasks tested.}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。