用生成模型重构强化学习，让离线模型更易在线优化。
Generative Actor Critic
- 将策略评估转为学习轨迹与回报的联合分布生成模型
- 在多个基准上实现更强离线性能和显著在线提升
- 适合需要高效迁移预训练模型的研究者
传统强化学习算法通常聚焦于估计或最大化期望回报，在用在线经验微调离线预训练模型时面临挑战。本文提出生成式演员-评论家（Generative Actor Critic, GAC），通过将策略评估重构为学习轨迹与回报的联合分布 $p(τ, y)$，并将策略改进转化为对该生成模型的灵活推理。我们基于带有连续潜变量计划向量的隐变量模型，设计了新的推理策略：利用潜变量优化实现利用，通过条件采样实现探索。在 Gym-MuJoCo 与 Maze2D 基准上的实验表明，即使在缺乏逐步奖励的情况下，GAC 仍展现出优于现有方法的离线性能及显著增强的离线到在线改进能力。
原文摘要 · Abstract (English)
Conventional Reinforcement Learning (RL) algorithms, typically focused on estimating or maximizing expected returns, face challenges when refining offline pretrained models with online experiences. This paper introduces Generative Actor Critic (GAC), a novel framework that decouples sequential decision-making by reframing \textit{policy evaluation} as learning a generative model of the joint distribution over trajectories and returns, $p(τ, y)$, and \textit{policy improvement} as performing versatile inference on this learned model. To operationalize GAC, we introduce a specific instantiation based on a latent variable model that features continuous latent plan vectors. We develop novel inference strategies for both \textit{exploitation}, by optimizing latent plans to maximize expected returns, and \textit{exploration}, by sampling latent plans conditioned on dynamically adjusted target returns. Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods, even in absence of step-wise rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。