用路径空间镜面下降法,让生成式策略在连续控制中更稳定高效。
Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schrödinger Bridge
- 将策略优化建模为路径空间的广义薛定谔桥问题
- 在14个任务上超越传统方法,避免了动作密度计算
- 适合需要高表达力策略的复杂连续控制场景
经典在线策略算法如PPO和镜面下降策略优化通过可计算的动作似然提供稳定的近端更新,但通常使用表达能力有限的高斯策略。基于扩散和流模型的生成式策略能提供更丰富的动作分布,但其自然定义的是多步去噪路径上的分布,终端动作密度常不可计算,与基于似然的在线策略更新存在不匹配。为此,我们提出GSB-MDPO(广义薛定谔桥镜面下降策略优化),将在线策略生成式优化建模为状态条件生成路径上的广义薛定谔桥问题,并通过镜面下降策略优化实现路径测度更新。核心思想是:广义薛定谔桥路径空间KL起到MDPO中的近端项作用,同时上界终端动作KL,从而无需显式计算终端动作似然即可直接控制执行动作分布。在Playground和Gym-MuJoCo的14个连续控制任务上的实验表明GSB-MDPO具有有效性,支持路径空间正则化作为多步生成式策略的合理近端更新机制。
原文摘要 · Abstract (English)
Classical on-policy algorithms such as PPO and mirror descent policy optimization provide stable proximal policy updates through tractable action likelihoods, but are typically instantiated with simple Gaussian policies whose expressiveness can be limited in complex continuous-control tasks. Generative policies based on diffusion and flow models provide more expressive action distributions, but they naturally define distributions over multi-step denoising paths whose terminal action density is often intractable, creating a mismatch with likelihood-based on-policy proximal updates. To address this mismatch, we introduce \textbf{GSB-MDPO} (\emph{Generalized Schrödinger Bridge Mirror Descent Policy Optimization}), which formulates on-policy generative policy optimization as a Generalized Schrödinger Bridge problem over state-conditioned generation paths and instantiates the resulting path-measure update through mirror descent policy optimization. The key insight is that the GSB path-space KL plays the role of the proximal term in MDPO while upper-bounding the terminal action KL, enabling direct control of the executed action distribution without explicit terminal action likelihood evaluation. Experiments on 14 continuous-control tasks across Playground and Gym-MuJoCo demonstrate the empirical effectiveness of GSB-MDPO and support path-space regularization as a principled proximal update for multi-step generative policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。