用状态分布差异约束策略更新，让离线数据更安全有效。
Divergence-Augmented Policy Optimization
- 引入行为策略与当前策略的状态分布Bregman散度，约束更新幅度。
- 在Atari数据稀缺场景下，性能超越现有主流强化学习算法。
- 适合需要高效利用离线数据的强化学习任务，如机器人控制。
深度强化学习中的策略优化方法需应对函数近似和离线数据复用问题。标准策略梯度方法对离线数据处理不佳，易导致过早收敛和不稳定。本文提出一种新方法，在复用离线数据时稳定策略优化。核心思想是引入生成数据的行为策略与当前策略之间的Bregman散度，确保更新小而安全。该散度基于两策略的状态分布计算，而非仅动作概率，形成散度增强公式。在Atari游戏上的实验表明，在数据稀缺场景下，本方法性能优于其他先进深度强化学习算法。
原文摘要 · Abstract (English)
In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。